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ABSTRACT 


The  research  area  of  this  thesis  is  Simulation  Based  Acquisition  (SBA) 
and  the  methods  in  which  it  has  been  implemented  into  the  Department  of 
Defense  (DoD).  Application  of  SBA  initiatives  relies  upon  the  use  of  modeling 
and  simulation,  among  other  methods,  to  effectively  use  scarce  resources  - 
funds,  manpower,  equipment  -  in  the  life  cycle  of  major  programs.  The  SBA 
initiative  has  been  used  in  the  Department  of  Defense  for  approximately  six 
years.  Accordingly,  program  managers  have  employed  the  techniques  of  SBA  to 
achieve  significant  advances  while  reducing  costs.  Thus,  those  resources  saved 
may  be  used  elsewhere  in  the  program.  Typically,  savings  can  be  realized  early 
in  the  program  life  in  terms  of  downstream  logistics.  Users  who  have  identified 
Operating  and  Sustainment  (O&S)  constraints  during  the  preparation  of  the 
Mission  Needs  Statement  should  examine  the  inclusion  of  SBA  during  concept 
exploration  and  system  development.  The  Department  of  Defense  has  provided 
guidance  on  the  SBA  initiative  and  the  methods  of  SBA  are  being  incorporated 
into  programs.  The  attempt  of  this  study  will  not  only  identify  the  key  points 
within  the  program  to  use  SBA,  but  also  how  to  best  employ  those  methods. 
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I.  INTRODUCTION 


A.  ABSTRACT 

The  research  area  of  this  thesis  is  Simulation  Based  Acquisition  (SBA) 

and  the  methods  in  which  it  has  been  implemented  into  the  Department  of 

Defense  (DoD).  The  application  of  SBA  initiatives  relies  upon  the  use  of 

modeling  and  simulation,  among  other  methods,  to  use  effectively  scarce 

resources  -  funds,  manpower,  equipment  -  in  the  life  cycle  of  major  programs. 

This  concept  has  been  defined  as: 

Simulation  Based  Acquisition  is  an  iterative,  integrated  product  and 
process  approach  to  acquisition,  using  modeling  and  simulation, 
that  enables  the  warfighting,  resource  allocation,  and  acquisition 
communities  to  fulfill  the  warfighter’s  materiel  needs,  while 
maintaining  Cost  As  an  Independent  Variable  (CAIV)  over  the 
system’s  entire  life  cycle  and  within  the  DoD’s  system  of  systems 
(Johnson,  et.  al .,  1997  -  1998). 


SBA  has  been  used  in  the  Department  of  Defense  for  approximately  six 

years.  Program  managers  have  employed  the  techniques  of  SBA  to  achieve 

significant  advances  while  reducing  costs.  Thus,  those  resources  saved  may  be 

used  elsewhere  in  the  program.  Typically,  savings  can  be  realized  early  in  the 

program  life.  The  U.S.  Army  has  recognized  this  point  and  is  actively  addressing 

it.  In  1998,  Ms.  Ellen  Purdy  (with  the  office  of  the  Secretary  of  the  Army  for 

Research,  Development,  and  Logistics  (since  renamed  Acquisition,  Logistics  and 

Technology  (ASA(AL&T)))  was  quoted  as  saying: 

What  costs  the  most  is  operation  and  support.  There  are  many 
opportunities  to  impact  the  cost  of  operating  and  supporting  a 
system  at  the  beginning  of  the  development  cycle.  By  the  time  we 
hit  Milestone  I,  we  have  locked  in  70  percent  of  the  cost  of 
operating  and  sustaining  a  system’  (AR  Today,  Nov/Dec  1998). 


Clearly,  the  Army  recognizes  the  impact  that  design  has  on  the  total 
ownership  cost  for  the  program.  Any  savings  that  can  be  attained  during  the 
development  of  the  system  translate  into  huge  downstream  funding 
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opportunities.  Users  who  have  identified  Operating  and  Sustainment  (O&S) 
constraints  during  the  preparation  of  the  Mission  Needs  Statement  should 
examine  the  inclusion  of  SBA  during  the  concept  exploration  and  system 
development. 

The  need  for  up-front  involvement  also  affects  the  risk  associated  with 

performance  as  can  be  seen  by  the  following  passage. 

The  SBA  process  offsets  increased  performance  risk  by  enabling 
the  user  to  become  a  member  of  the  design  team  and  to  influence 
the  design  much  earlier  than  the  current  process  allows;  and  this 
provides  rapid  feedback  to  the  design  team  by  enabling  them  to 
perform  “what  if”  analyses,  or  iterations,  on  hundreds  of  different 
designs  (AR  Today,  Oct/Nov  2000). 

Using  modeling  or  simulation  is  a  powerful  tool  for  the  program  manager 
who  seeks  to  reduce  performance  risk,  incorporate  concurrent  design  changes, 
or  examine  the  possibility  of  using  off  the  shelf  commercial  items.  Each  of  these 
possibilities  can  be  analyzed  in  the  Modeling  and  Simulation  (M&S)  process  and 
have  an  immediate  determination  as  to  its  usefulness. 

The  Department  of  Defense  has  provided  guidance  on  the  SBA  initiative 
and  the  methods  of  SBA  are  being  incorporated  into  programs.  However,  it  is 
not  specifically  clear  to  program  personnel  on  how  to  best  utilize  these 
techniques.  The  attempt  of  this  study  will  not  only  identify  the  key  points  within 
the  program  to  use  SBA,  but  also  how  to  best  employ  those  methods. 

B.  SCOPE  OF  THE  THESIS 

It  is  the  intent  of  the  thesis  to  concentrate  on  Simulation  Based  Acquisition 
(SBA)  and  its  impact  on  military  programs.  The  basic  premise  of  SBA  is  to 
reduce  initial  costs  through  modeling  and  simulation.  These  cost  savings  can 
then  be  driven  back  into  the  program  in  some  form.  Oftentimes,  these  funds  can 
be  used  for  additional  units  in  production,  spares  and  repairs,  or  effective 
logistics  management. 

The  thesis  research  will  include  a  survey  that  will  concentrate  on  Navy 
programs  across  all  Acquisition  Category  (ACAT)  programs.  All  data  collected 
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will  be  analyzed  through  the  narrative  response  provided  by  the  respondents. 
The  thesis  will  attempt  to  identify  the  points  in  a  program  that  effectively  use 
different  forms  of  SBA  to  achieve  cost  avoidance  or  technical  advances.  In  doing 
so,  the  result  would  be  a  more  effective  use  of  funding  for  additional 
requirements  within  the  same  acquisition. 

The  principal  question  to  be  answered  through  the  study  can  be  stated  as: 
“Are  there  specific  times  within  the  life  cycle  of  a  major  program  when  it  is  in  the 
best  economic  interest  of  the  Program  Manager  to  use  the  methods  of  Simulation 
Based  Acquisition?”  For  example,  can  Program  Managers  or  other  key  leaders 
identify  SBA  methods  that  can  be  employed  at  the  onset  of  Concept 
Development,  System  Development  and  Demonstration,  and  Production  and 
Development? 

C.  THESIS  ORGANIZATION 

In  this  thesis,  the  paper  will  attempt  to  identify  economic  breakpoints  in  a 
program  where  employing  SBA  techniques  will  easily  provide  funding  benefits. 
One  may  assume  that  such  breakpoints  would  occur  at  each  Milestone  decision. 
In  attempting  to  answer  this  question,  the  thesis  will  identify  and  analyze  several 
supporting  sub-questions.  These  queries  form  the  underlying  essence  of  each 
chapter.  Those  same  questions  may  be  documented  as  the  following: 

1 .  What  is  the  background  and  history  of  Simulation  Based  Acquisition? 

2.  How  has  the  Department  of  Defense  provided  guidance  on  Simulation 
Based  Acquisition? 

3.  Have  there  been  SBA  success  stories  in  the  Department  of  Defense? 

4.  What  data  can  be  reviewed  to  determine  the  ability  of  SBA  to  provide 
economic  breakpoints?  What  are  the  results  of  the  data  analysis  and 
how  may  these  results  be  applied  to  SBA? 

5.  What  conclusions  may  be  drawn  from  the  preceding  information  that 
might  be  applied  to  DoD  programs? 
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D.  METHODOLOGY 

The  thesis  uses  standard  literary  research  for  addressing  the  first  three 
issues  listed  above.  In  addressing  the  hard  data,  the  study  will  make  use  of  a 
survey  to  derive  information  on  the  use  of  SBA  in  Navy  Programs.  The  survey 
will  be  designed  to  identify  the  degree  that  Program  Managers  have  used  SBA  in 
the  past  and  how  SBA  is  incorporated  into  the  overall  program. 

E.  BENEFITS  OF  THE  STUDY 

The  thesis  is  timely  in  the  sense  that  the  Department  of  Defense  has 
created  an  initiative  to  use  SBA  throughout  programs.  However,  there  is  not 
clear  direction  on  how  to  adequately  implement  SBA.  This  study  is  expected  to 
benefit  Program  Managers  of  Major  Weapons  programs  as  well  as  provide 
guidance  to  Program  Managers  of  smaller  acquisition  programs. 

The  Soldier  and  Biological  Chemical  Command  (SBCCOM)  Modeling  and 
Simulation  (M&S)  Team  has  expressed  interest  in  this  study.  The  SBCCOM 
M&S  Team  provides  simulations  across  Services  for  high  fidelity  representation 
of  battlefield  environments.  These  simulations  are  typically  used  in  soldier  and 
unit  level  training  to  improve  efficiency  on  the  battlefield.  However,  the  SBCCOM 
M&S  Team  also  can  provide  to  its  customers  the  ability  to  test,  though  modeling 
and  simulation,  of  systems.  Use  of  the  data  and  conclusions  from  the  thesis 
would  be  employed  by  the  M&S  Team  to  carry  forward  to  all  levels  of  Program 
Managers  and  headquarters.  This  information  would  then  be  disseminated  to 
provide  a  better  understanding  of  the  essentials  of  SBA  techniques  and  how  SBA 
can  improve  the  acquisition  program. 
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II.  BACKGROUND 


A.  SCOPE 

The  use  of  Simulation  Based  Acquisition  (SBA)  throughout  the 
Department  of  Defense  (DoD)  is  a  tool  that  many  Program  Managers  (PM)  have 
employed  in  military  programs.  Succinctly  stated,  SBA  is  the  use  of  simulation 
(M&S)  techniques  and  tools  to  leverage  resources  in  the  program. 

There  has  been  significant  work,  by  the  Government  and  private  sector,  in 
bringing  the  elements  of  SBA  to  program  managers.  This  research  will  use  the 
published  guidance  and  industry  studies  as  a  base  from  which  to  illustrate  the 
tenets  of  SBA.  Accordingly,  a  lesser  intent  of  this  thesis  is  to  serve  as  a  basis  for 
future  studies. 

B.  WHAT  SBA  IS 

Simulation  Based  Acquisition  can  be  a  difficult  concept  to  express  in  one 
concise  statement.  As  such,  SBA  may  be  thought  of  as  an  amalgamation  of 
various  Modeling  and  Simulation  (M&S)  methods  throughout  the  acquisition  life 
cycle.  For  example,  the  use  of  Computer  Aided  Design  (CAD)  for  developing  a 
revised  technical  drawing  is  as  much  a  part  of  SBA  as  is  a  training  simulator 
used  by  soldiers  in  the  field  or  a  Distributed  Interactive  Simulation  (DIS)  used  by 
numerous  activities  concurrently. 

The  concept  of  SBA  is  more  of  a  mindset  and  philosophy  than  a  tangible 
set  of  steps  in  place  at  specific  points  in  the  program’s  life.  A  definition  has  been 
set  forth  as  “An  acquisition  process  in  which  DoD  and  Industry  are  enabled  by 
robust,  collaborative  use  of  simulation  technology  that  is  integrated  across 
acquisition  phases  and  programs  (DMSO,  2000).”  The  point  paper  continued  by 
identifying  three  goals  of  SBA.  These  goals  are: 

•  Substantially  reduce  the  time,  resources,  and  risk  associated  with  the 
entire  acquisition  process. 
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•  Increase  the  quality,  military  worth,  and  supportability  of  fielded 
systems  while  reducing  total  ownership  costs  throughout  the  total  life 
cycle. 

•  Enable  Integrated  Product  and  Process  Development  (IPPD)  across 
the  entire  acquisition  life  cycle.  (DMSO,  2000) 

This  definition  conveys  the  crucial  elements  of  SBA  and  its  impact  on  the 
life  cycle  of  the  program.  The  elements  are  risk  reduction,  cost  reduction,  and 
ownership  of  the  program.  Through  the  use  of  modeling  and  simulations, 
program  managers  can  leverage  scarce  resources  while  improving  development 
and  fielding. 

Caution  is  made  in  the  point  paper  of  what  SBA  is  not.  SBA  is  not  a 
waiver  of  common  sense  and  required  activities.  Specifically,  SBA  should  not  be 
used  as  a  substitute  for  proper  planning,  insightful  engineering,  allowing  the 
modeling  to  dictate  program  direction,  or  abdicating  the  responsibility  for  the 
program  (DMSO,  2000).  Therefore,  the  PM  must  continue  to  remain  as  the 
program’s  most  vigorous  advocate  while  encouraging  the  program  team  to  buy 
into  total  ownership  costs. 

Another  DoD  paper  on  SBA  is  more  definitive  in  what  SBA  is  not.  The 
paper  states  that  SBA  should  not  be: 

•  A  replacement  for  good  systems  engineering; 

•  Having  simulations  make  the  decisions; 

•  Giving  all  information  to  everyone  and  letting  everyone  see 
everything  you  do; 

•  The  loss  of  security  and  proprietary  advantage; 

•  The  loss  of  responsibility,  authority,  and/or  accountability; 
and, 

•  Just  using  M&S  in  an  acquisition  program  (NARO,  2001) 

Clearly,  the  implication  here  is  that  SBA  is  a  tool  to  be  effectively  wielded 
by  program  managers  in  advancing  their  programs.  PMs  that  rely  upon  SBA  and 
ignore  other  factors  (not  a  replacement  for  good  systems  engineering)  may  find 
that  SBA,  in  such  a  case,  might  actually  be  a  detriment  to  their  program.  This  is 
analogous  to  using  a  periscope  to  look  only  forward  and  not  use  other  detection 
and  navigational  aides  in  plotting  future  courses. 


6 


Indeed,  SBA  should  almost  be  considered  a  sub-function  of  systems 
engineering  with  its  ability  to  rapidly  simulate  different  configurations  of  a  system 
dependant  upon  intended  system  use  or  proposed  doctrine.  Ultimately,  the 
program  manager  must  be  the  final  authority  on  program  issues  rather  than  a 
simulation  generated  decision. 

C.  DOD  BACKGROUND 

In  1995,  the  Director,  Operational  Test  and  Evaluation  (DOT&E),  released 
his  annual  report.  Contained  within  the  report  was  a  discussion  of  Secretary  (of 
Defense)  Perry’s  theme’s  for  the  future.  One  theme  was  that  modeling  and 
simulation  (M&S)  needed  to  be  used  more  effectively  in  acquisitions  and  program 
life  cycles.  It  may  be  inferred  from  the  annual  report  that  M&S  is  not  inexpensive, 
but  that  the  initial  cost  is  recouped  by  more  robust  designs,  less  hardware 
needed  for  testing,  and  effective  training  for  personnel.  The  report  also  quoted 
the  USD(A&T),  Dr.  Paul  Kaminski,  as  saying,  “This  means  our  underlying 
approach  will  be  to  model  first,  simulate,  then  test,  and  then  iterate  the  test 
results  back  into  the  model.”  The  underlying  requirement  was,  therefore,  to  have 
M&S  be  a  part  of  each  Test  and  Evaluation  Master  Plan  (TEMP).  With  this 
emphasis,  programs  would  have  attributed  more  integration  to  modeling  and 
simulation  activities  such  that  the  program  derives  these  robust  designs  (DOTE 
FY95  Annual  Report). 

In  1998,  the  Undersecretary  of  Defense  Acquisition  and  Technology 
(USD(A&T))  spoke  before  the  National  Defense  Industrial  Association.  In  his 
remarks,  Undersecretary  Gansler  stated  that  there  are  three  significant 
challenges  before  the  Department  of  Defense.  These  are  to  develop/deploy 
major  new  systems,  to  support  those  systems,  and  do  this  at  a  lower  cost  and 
reduced  cycle  time  (Gansler,  1998).  Part  of  the  answer  to  this  challenge  is  the 
implementation  of  SBA  in  the  major  (and  minor)  military  programs.  Simulation 
Based  Acquisition  easily  meets  these  demands  through  the  use  of 
modeling/simulation,  defining  program  requirements  resulting  from  those 
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simulations,  and  having  the  personnel  be  dedicated  to  the  success  of  the 
program  overall. 

Indeed,  Mr.  Gansler  went  on  to  state  that  modeling  and  simulation  must 

be  a  greater  part  in  testing  and  evaluation  (Gansler,  1998).  Additionally,  such 

use  of  M&S  in  testing  must  be  validated,  though,  prior  to  implementation. 

Therefore,  the  resources  that  program  managers  must  judiciously  employ  are 

applied  effectively  through  validated  M&S  activities.  The  virtual  testing  does 

reduce  the  program  risk  while  reducing  program  cost.  It  may  be  inferred  that  Mr. 

Gansler  implored  program  managers  to  use  SBA  proactively  to  offset  risk,  while 

defining  a  robust  and  dynamic  system  design. 

The  DoD  document,  DoD  5000.1  states  that, 

Program  managers  shall  plan  and  budget  for  effective  use  of 
modeling  and  simulation  to  reduce  time,  resources,  and  risk 
associated  with  the  entire  acquisition  process,  increase  the  quality, 
military  worth,  and  supportability  of  fielded  systems;  and  reduce 
total  ownership  costs  throughout  the  system  life  cycle.  (DoDD 
5000.1,  The  Defense  Acquisition  System,  (Incorporating  Change  1, 
January  4,  2001 )  23  October  2000) 

The  DoDD  5000  series  has  been  cancelled  and  replaced  with  interim 
guidance  from  Undersecretary  of  Defense  Paul  Wolfowitz.  By  memorandum 
dated  30  Oct  02,  the  guidance  cancelled  the  5000  series  and  provided  two 
attachments.  The  first  attachment  provides  the  program  manager  with  policies 
that  are  simple  and  flexible  to  encourage  maximum  autonomy  for  the  PM.  In  the 
attachment,  the  guidelines  discuss  numerous  aspects  of  program  management 
including  cost  realism,  cost  sharing,  and  program  affordability.  More  specifically, 
the  guidelines  instruct  the  program  manager  to  define  total  ownership  costs  (and 
those  drivers),  identify  cost  risks  before  contract  award,  and  that  undue  risks  are 
not  placed  upon  the  contractors.  (Wolfowitz,  02) 

Therefore,  to  keep  in  compliance  with  this  most  basic  requirement  and  to 
manage  their  systems  in  a  smarter  fashion,  it  is  axiomatic  for  the  PM  to  employ 
modeling  and  simulation,  as  a  minimum,  in  their  program. 
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The  Army  has  embraced  the  concept  of  Simulation  Based  Acquisition 
through  its  propagation  of  Simulation  and  Modeling  for  Acquisition, 
Requirements,  and  Training  (SMART)  (Price,  2002).  The  intent  of  SMART  is  to 
improve  higher  readiness  levels  through  reduction  of  life  cycle  costs. 

The  Navy,  as  early  as  1994,  sought  to  incorporate  modeling  and 
simulation  into  their  policy.  A  Modeling  and  Simulation  Program  was  established 
to  oversee  the  development  and  use  of  models  and  simulations.  The  guidance 
went  on  to  discuss  interoperability  issues,  joint  configuration  concerns,  and 
define  how  coordination  of  plans  shall  occur.  (DONMSP)  (Navy,  1994).  In  1998, 
the  Navy  moved  forward  with  further  instruction  regarding  Simulation  Based 
Acquisition.  An  Acquisition  Reform  Executive  was  established  as  the  lead  for 
SBA  issues  and  initiatives.  The  intent  was  to  meet  the  Defense  System 
Affordability  Council  goals  of... 

...fielding  high  quality  defense  systems  quickly,  and  supporting  them 

responsively,  while  lowering  total  ownership  cost  (Navy,  1998). 

The  Navy,  like  the  other  Services,  was  moving  to  incorporate  the  use  of 
SBA  into  its  programs. 

D.  INDUSTRY’S  VIEWPOINT 

In  1995,  the  American  Defense  Preparedness  Association  (ADPA) 
proposed  a  study  of  SBA  and  its  role  in  the  military  and  industrial  environment. 
The  study  defined  SBA  as  having  three  separate  elements  in  its  composition, 
environment,  and  use.  The  first  element  is  a  forum  in  which  systems  engineering 
techniques  are  proactively  utilized  through  the  implementation  of  advanced 
technology  systems.  The  system  engineering  techniques  are  most  often  used  in 
the  capacity  of  design  efficacy  (better  designs  and  less  manpower  to  design)  and 
life  cycle  tools. 

The  second  element  identified  in  the  study  is  the  use  of  better  design  data 
for  implementation  in  the  life  cycle  acquisition  process.  Such  design 
improvements  can  be  leveraged  in  acquisitions  through  more  efficient  design  and 
fielding  requirements.  Furthermore,  the  Performance  Specification,  as  it  changes 
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over  the  life  of  the  program,  will  be  more  accurate  based  upon  the  knowledge 
and  data  gathered  through  the  improved  design  process. 

The  final  element  ADPA  found  in  SBA  is  the  social  norm  of  the  personnel 
involved  in  the  process.  The  participants  in  the  program  must  be  willing  to 
accept  the  values  imposed  and  be  encouraged  to  be  creative  in  their  respective 
areas.  Additionally,  the  participants  are  encouraged  to  think  outside  the  box  and 
use  the  data  from  SBA  to  leverage  new  concepts  and  ideas.  (ADPA,  1996). 
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III.  GUIDANCE 


Should  one  desire  to  find  a  specific  source  or  definition  for  Simulation 
Based  Acquisition  in  Government  documentation,  one  would  undoubtedly 
encounter  a  surfeit  of  guidance,  policy,  and  regulatory  information.  Before 
attempting  to  understand  guidance  that  controlled  Service  implementation  of 
SBA,  one  should  appreciate  the  concepts  of  SBA.  This  chapter  will  set  forth  the 
definition  of  SBA  as  well  as  illustrating  the  policy  of  the  Defense  Department. 

A.  DEPARTMENT  OF  DEFENSE 

In  1995,  based  upon  the  direction  of  the  Clinton  Administration,  the 
Department  of  Defense  identified  ways  in  which  the  acquisition  process  could  be 
streamlined.  The  prevailing  environment  of  doing  more  with  fewer  resources 
mandated  PMs  to  critically  evaluate  their  programs.  Accordingly,  DoD  moved 
assertively  towards  better  use  of  resources  in  military  programs.  This  same 
environment  motivated  higher  headquarters  staffs  to  clarify  methods  in  which 
those  same  resources  could  be  leveraged  as  a  form  of  policy.  The  genesis  of 
Simulation  Based  Acquisition  was  created  from  within  this  atmosphere. 

1.  Department  Of  Defense  Directive  (DoDD)  5000.59,  DoD  Modeling 
and  Simulation  (M&S)  Management 

This  directive,  issued  04  Jan  94,  provided  initial  guidance  in  the 
management  of  modeling  and  simulation.  The  directive  sets  forth  the 
requirements  and  stands  up  the  Executive  Council  for  Modeling  and  Simulations 
(EXCIMS).  The  document  also  stands  up  the  Defense  Modeling  and  Simulation 
Office  (DMSO).  Under  DMSO,  the  Modeling  and  Simulation  Information  Analysis 
Center  (MSIAC).  Numerous  policies  are  established  in  this  document  including 
the  following: 

•  Setting  an  organizational  structure  for  the  management  and 
administration  for  DoD  M&S  issues; 

•  Developing  a  Master  Plan  for  M&S; 
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•  Investing  in  the  resources  required  for  M&S  including 
enhancements,  common  tools  and  databases,  and  establishing 
standards/protocols  for  data  exchange; 

•  Implementing  a  M&S  Information  Analysis  Center  (MSIAC); 

•  Assigning  common  and  general  use  applications  to  a  specific 
Component  for  responsibility; 

•  Directing  that  the  Services  shall  establish  Verification, 
Validation,  and  Accreditation  (VV&A)  policies  for  the  M&S 
applications  managed  by  the  Component; 

•  Requiring  that  all  the  applications  in  use  by  DoD  decision 
makers  must  be  accredited;  and, 

•  Requiring  that  all  applications  must  conform  with  DoDD  8320.1 
“DoD  Data  Administration  (DoDD  5000.59,  1994) 

The  directive  also  stated  that  the  Undersecretary  of  Defense  for 
Acquisition  and  Technology  establish  the  EXCIMS.  The  mission  of  EXCIMS  is  to 
be  the  advocate  for  M&S  in  the  Department  of  Defense.  The  Council  will  advise 
the  USD(A&T)  on  the  uses  of  M&S,  oversee  M&S  development,  improve 
communications,  invest  in  M&S  applications,  and  promote  cooperative  research 
within  the  Services. 

The  EXCIMS  Charter  mandates  that  this  council  advises  and  assists  the 
Undersecretary  of  Defense  for  Acquisition  and  Technology  (now  USD  (AL&T))  in 
the  aspects  of  Modeling  and  Simulation  throughout  DoD.  The  Charter  goes  on  to 
state  eight  functions  which  EXCIMS  must  undertake  in  order  to  achieve  this 
assistance.  These  are: 

•  Oversees  development  of  DoD  M&S  policies,  plans,  programs, 
publications,  and  procedures. 

•  Encourages  improved  communication  and  coordination  among 
DoD  M&S  activities. 

•  Identifies  investments  in  M&S  that  have  high  value  return  in 
fulfilling  DoD  requirements,  or  that  fill  gaps  in  M&S  capabilities. 

•  Promotes  joint  and  cooperative  research,  development, 
acquisition,  and  operation  of  M&S  systems,  technologies,  and 
capabilities  among  DoD  components. 

•  Recommends  DoD  M&S  goals,  objectives,  and  an  investment 
strategy  and  plan  to  achieve  them. 

•  Recommends  DoD  components  for  designation  as  DoD  M&S 
Executive  Agents  for  general  use  M&S  applications,  as  needed. 
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•  Acts  as  an  Executive  Steering  Committee  for  DoD  general  use 
M&S  applications  for  which  Executive  Agents  have  been 
appointed. 

•  Fosters  programs  to  develop  and,  where  applicable,  implement 
DoD  M&S  interoperability  standards  and  protocols. 

Under  these  functions,  the  EXCIMS  is  the  lead  group  for  M&S  across 
DoD.  One  of  the  most  critical  functions  that  EXCIMS  is  tasked  with  is  the 
interoperability  issue.  As  Defense  programs  become  more  advanced  with 
technology,  the  ability  for  systems  to  communicate  with  each  other  diminishes. 
This  holds  true  for  modeling  and  simulation  as  well.  It  is  critical  that  models  and 
simulations  be  able  to  be  used,  simultaneously,  across  and  within  all  Services. 
By  doing  so,  then  digital  mock-ups  of  battlefield  conditions  can  be  used  for 
training,  software  used  for  virtual  training  on  missiles  might  be  used  for  the 
platforms  that  would  carry  the  missiles,  and  long  term  logistical  concerns  for  one 
Service  can  be  identified  early  in  the  program  life. 


2.  Department  Of  Defense  Instruction  (DoDI)  5000.61,  DoD  Modeling 

and  Simulation  (M&S)  Verification,  Validation,  and  Accreditation 

(W&A) 

The  intent  of  this  instruction  is  to  build  off  the  aforementioned  DoDD 
5000.59  regarding  modeling  and  simulation.  A  crucial  element  of  developing  a 
model  or  simulation  is  the  determination  that  the  simulation  is  appropriate,  useful, 
and  approved  by  the  proper  chain  of  command. 

According  to  the  VV&A  Recommended  Practices  Guide  Glossary,  the 
following  definitions  should  be  used  for  these  terms: 

•  Verification:  The  process  of  determining  that  a  model  or 
simulation  faithfully  represents  the  developer’s  conceptual 
description  and  specification.  Verification  evaluates  the  extent 
to  which  the  model  or  simulation  has  been  developed  using 
sound  and  established  software  and  system  engineering 
techniques. 

•  Validation:  The  process  of  determining  the  degree  to  which  a 
model  and  its  associated  data  are  an  accurate  representation  of 
the  real  world  from  the  perspective  of  the  intended  uses  of  the 
model. 
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•  Accreditation:  The  official  certification  that  a  model,  simulation, 
or  federation  of  models  and  simulations  and  its  associated  data 
are  acceptable  for  use  for  a  specific  purpose  (VV&A  Guide, 
2001). 


DoDI  5000.61  sets  forth  the  guidance  for  the  use  and  implementation  of 
VV&A  within  the  Department  of  Defense.  In  this  Instruction,  the  following  points 
are  set  forth: 

•  DoD  Components  shall  establish  VV&A  policies  and  procedures 
for  every  M&S  they  develop  and  manage. 

•  M&S  used  to  support  the  major  DoD  decisionmaking 
organizations  and  processes... shall  be  accredited  for  that  use 
by  the  DoD  Component  sponsoring  the  application.  Likewise, 

M&S  used  for  joint  training  and  joint  exercises  shall  be 
accredited  for  that  purpose  by  the  application  sponsor. 

•  Each  DoD  Component  shall  be  the  final  authority  for  validating 
representations  of  its  own  forces  and  capabilities  in  joint-, 
general-,  and  common-use  M&S  applications  and  shall  be 
responsive  to  the  other  DoD  Components  ensure  they  are 
appropriate  represented  (DoDI  5000.61). 


The  intent,  therefore,  is  to  have  models  and  simulations  that  are  accurate, 

highly  representative,  and  re-usable.  The  Modeling  and  Simulation  Information 

Analysis  Center  (MSIAC)  states  that  SBA  is... 

The  aggressive,  comprehensive  application  and  sharing  of  mature 
advancements  in  information  technology  such  as  distributed 
networking,  multi-user  computer  environments,  database 
management  systems,  and  particularly  advanced  modeling  and 
simulation  (M&S)  tools,  including  commercial  product  development 
automation  tools,  (e.g.  CAD,  ERP),  HLA-based  distributed 
simulation,  and  interactive  virtual  reality  (MSIAC,  00). 


The  MSIAC,  which  is  sponsored  by  both  the  Defense  Modeling  and 

Simulation  Office  (DMSO)  and  the  Defense  Technical  Information  Center  (DTIC), 

has  offered  an  excellent  working  definition  for  SBA  as  follows: 

An  acquisition  process  in  which  DoD  and  Industry  are  enabled  by 
robust,  collaborative  use  of  simulation  technology  that  is  integrated 
across  acquisition  phases  and  programs  (MSIAC,  00). 
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This  guidance  dovetails  with  that  set  forth  in  DoDI  5000.1  to  use  modeling 
and  simulation  to  reduce  costs  (para  4.5.4)  while  increasing  quality.  This 
definition  paper  went  on  to  state  that  a  key  aspect  of  SBA  is  the  use  and  sharing 
of  mature  advancements  in  information  technology.  However,  the  technology 
also  must  be  able  to  be  used  by  numerous  programs,  sometimes  at  the  same 
time  (in  the  case  of  a  distributed  interactive  simulation  [DIS]). 

3.  Simulation,  Test,  and  Evaluation  Process  (STEP) 

The  Department  of  Defense  has  established  guidelines  for  the 

implementation  of  SBA  into  military  programs.  These  guidelines,  entitled 

Simulation,  Test,  and  Evaluation  Process  (STEP),  are  designed  to  assist  the 

program  manager  in  the  application  of  SBA  into  their  programs.  In  the  guide  to 

implementing  STEP,  the  Honorable  Philip  E.  Coyle  (Director  of  Operational  Test 

and  Evaluation)  and  Dr.  Patricia  A.  Sanders  (Director  of  Defense  Research  and 

Engineering,  OSD)  have  stated  the  purpose  of  the  process  as  follows: 

The  Simulation  Test  and  Evaluation  Process  (STEP)  is  one  that 
integrates  both  simulation  and  test  for  the  purpose  of  interactively 
evaluating  and  improving  the  design,  performance,  joint  military 
worth,  survivability,  suitability,  and  effectiveness  of  systems  to  be 
acquired  and  improving  how  those  systems  will  be  used.  (Coyle 
and  Sanders,  1997) 

The  clarity  of  this  statement  to  the  Program  Manager  is  critical.  The  PM, 
through  the  use  of  STEP,  needs  to  actively  incorporate  simulation  and  test 
techniques  in  the  program.  By  doing  so,  the  whole  program  will  be  dynamic 
throughout  its  life  cycle  through  operative  improvements.  The  STEP  Guidelines 
go  on  to  state: 

The  Department’s  vision  is  to  have  an  acquisition  process  that  is 
enabled  by  the  robust,  collaborative  use  of  simulation  technology 
that  is  integrated  across  acquisition  phases  and  programs.  The 
goals  of  Simulation  Based  Acquisition  are  to: 
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•  Substantially  reducing  the  time,  resources,  and  risk  associated 
with  the  acquisition  process, 

•  Increasing  the  quality,  military  utility,  and  supportability  of 
systems  fielded,  while  reducing  their  total  ownership  costs,  and 

•  Facilitating  Integrated  Product  and  Process  Development 
(IPPD)  across  the  full  acquisition  life  cycle.  (Coyle  and  Sanders, 

1997) 

Traditionally,  the  method  in  which  program  or  system  problems  are 

addressed  had  been  through  a  test-fix-test  process.  The  STEP  process  has  the 

program  manager  using  a  model-simulate-fix-test-iterate  progression.  In  this 

approach,  the  system  incorporates  design  improvements  and  problematic  fixes 

through  simulation.  The  process  contains  more  than  one  model.  As  the  system 

progresses  through  the  acquisition  cycle,  models  are  built  to  represent  the 

system  at  the  current  level  of  development.  Therefore,  as  progress  is  made, 

models  are  developed  for  interfaces,  environments,  and  future  design  potential. 

Coyle  and  Sanders  went  on  to  describe  how  STEP  could  be  applied  throughout 

the  old  acquisition  cycle  (CE  -  Production). 

Analysis  of  Mission  Need:  Models  and  simulations  offer  a  way  to 
quantify  the  shortfalls  in  numbers  and  type  of  systems  for  force 
structure  analysis,  to  quantify  the  consequences  for  a  wide  variety 
of  scenarios,  and  to  identify  thresholds  of  operational  significance 
to  the  accomplishment  of  national  military  objectives. 

Concept  Exploration:  During  this  phase,  an  Analysis  of 
Alternatives  (AOA)  is  developed  to  aid  and  document 
decisionmaking  by  providing  insight  into  the  relative  advantages 
and  disadvantages  of  the  options  being  considered  to  meet  a 
mission  need. 

Program  Definition  and  Risk  Reduction:  During  the  Program 
Definition  and  Risk  Reduction  (PDRR)  phase,  STEP  can  provide: 
early  insight  into  the  reliability,  availability,  and  maintainability 
(RAM)  of  the  proposed  system;  information  to  support  the 
assessment  of  risk;  information  from  ergonomic  models  to  support 
maintainability;  information  on  physics  of  failure;  and  data  on  the 
human-machine  interface.  Predictions  from  engineering-level  M&S 
are  also  used  as  a  the  basis  for  representing  the  system’s 
performance  in  engagement-level  M&S 

Engineering  and  Manufacturing  Development:  During  the 
Engineering  and  Manufacturing  Development  (EMD)  phase,  the 
most  promising  design  approach  is  translated  into  a  stable, 
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interoperable,  producible,  supportable,  and  cost-effective  design. 
This  is  when  models,  simulations  and  tests  are  used  to: 

(1 )  verify  the  system’s  design; 

(2)  confirm  that  design  risks  have  been  controlled; 

(3)  certify  readiness  for  operational  testing;  and 

(4)  evaluate  the  system’s  operational  effectiveness, 
suitability,  and  survivability. 

Feed  back  from  the  tests  to  simulations  is  not  only  required  for 
verification,  validation,  and  accreditation  (VV&A),  but  also  to  allow 
progressive  improvement  in  the  M&S.  In  this  manner  M&S  can 
support  the  tests  with  performance  predictions  for  use  in  planning 
future  tests  and  in  risk  assessment. 

Production,  Fielding/Deployment,  and  Operational  Support: 

The  process  of  updating  models  and  data  sets  is  critical  in  resolving 
design  problems  after  fielding  and  in  making  modifications  to  the 
system  throughout  its  life  cycle.  (Coyle  and  Sanders,  1997) 


Figure  1  shows  the  relationship  between  the  previous  5000  series 
acquisition  cycle  and  the  life  cycle  set  forth  in  Attachment  2  to  the  30  Oct  02 
Deputy  Secretary  of  Defense  Paul  Wolfowitz  memorandum.  Through  the 
comparison  below,  the  above  uses  for  STEP  can  easily  be  transferred  to  the  new 
cycle  model  (the  upper  scale). 
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Figure  1 

A  Comparison  Between  the  5000  Series  Timeline  and  the  Revised  Model 

The  use  of  M&S  in  system  development  is  critical  in  order  for  both  the 
system  design  as  well  as  the  model/simulation  to  mature  appropriately.  Figure  2, 
is  a  representation  of  how  modeling  and  simulation  flow  upward  to  campaign 
specific  goals.  The  STEP  guidelines  describe  this  flow,  as  did  the  PDRR 
reference  above: 
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Campaign 


Engineering-level  models  often  involve  the  use  of  three- 
dimensional  CAD/CAM/CAE,  or  computational  fluid  dynamics  or 
other  computational  approaches  to  consider  such  effects  as 
signatures,  extreme  environments,  mobility,  and  fatigue. 
Engagement-level  models  are  used  to  explore  such  issues  as 
end-game  lethality,  firepower,  manufacturing,  and  producibility. 
Mission/battle-level,  system-on-system  models  support 
analyses  of  system  performance,  platform  engagement, 
survivability,  and  mobility. 

Theater/campaign-level,  force-on-force  models  provide  insights 
into  a  system’s  contribution  to  force  effectiveness,  as  well  as  force 
structure  and  logistics  requirements.  (Coyle  and  Sanders,  1997) 


So  the  incorporation  of  STEP  into  the  programs  should  improve  system 

design  at  all  points  in  the  acquisition  life  cycle.  Additionally,  STEP  addresses  the 

use  of  SBA,  by  way  of  M&S,  from  engineering  design  to  campaign  systems.  As 

the  Department  of  Defense  promulgated  the  use  of  SBA  in  programs,  each 

Component  developed  their  own  policies  on  SBA  and  Modeling  and  Simulation. 
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The  following  sections  of  this  chapter  address  the  Components’  actions  in 
providing  direction  on  SBA. 


B.  DEPARTMENT  OF  THE  ARMY 

The  US  Army  has  established  guidance  on  the  management  and 
administration  of  modeling  and  simulation.  The  Army  Regulation  (AR)  directs  the 
Deputy  Chief  of  Staff  for  Operations  and  Plans  (DCSOPS)  to  establish  the  Army 
Modeling  and  Simulation  Office  (AMSO).  AR  5-1 1  goes  on  to  state  that: 

The  Army’s  philosophy  is  that  M&S,  to  include  Simulators,  are  not 

an  end  unto  themselves,  but  a  critical  set  of  closely  related  tools 

which  contribute  to  the  accomplishment  of  Army  missions  (AR  5-1 1 , 

Jul  97). 

The  guidance  further  defines  four  elements  of  M&S  management.  These 
are:  develop  policy;  establish  standards;  prioritize  and  integrate  requirements 
and  investments;  and,  direct  research  and  technology. 

The  use  of  M&S  in  Army  programs  is  divided  into  three  distinct  functional 
groups.  These  are: 

•  Training,  Exercises  and  Military  Operations  (TEMO)  -  This  area 
includes  individual  and  collective  training;  joint  and  combined 
exercises;  mission  rehearsal;  and,  operations  planning.  This  domain 
would  use  system  and  training  simulators  to  accomplish  these  tasks. 

•  Advance  Concepts  and  Requirements  (ACR)  -  The  domain  includes 
force  design;  operational  requirements;  and,  warfighting  experiments. 
Reconfigurable  simulators  and  constructive  models  are  applied  to 
these  tasks. 

•  Research,  Development,  and  Acquisition  (RD&A)  -  Basic  applied 
research;  weapons  system  development;  and,  test  and  evaluation 
make  up  this  domain.  System  prototypes  and  engineering/physics 
models  are  used  to  complete  these  tasks.  (AR  5-1 1 ) 

The  US  Army  Modeling  and  Simulation  Office  is  the  lead  for  Simulation 
and  Modeling  for  Acquisition,  Requirements,  and  Training  (SMART).  This 
process  has  been  described  as  a  revolution... 

•  in  process:  replacing  a  traditional  linear  acquisition  cycle  with  an 
integrated,  iterative  process  in  which  simulation  and  virtual 
prototyping  replaces  hardware  prototyping  in  early  cycles. 
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•  in  collaboration  technology:  seamless  data  exchange  will 
support  the  joining  of  a  distributed  IPT  into  a  common  data 
framework. 

•  in  simulation  technology:  extending  the  precepts  of  simulation 
across  all  phases  of  the  acquisition  process,  across  all 
functional  domains,  and  across  all  levels  of  fidelity. 

•  in  culture:  providing  the  Acquisition  Manager  with  the  analytic 
and  evaluation  tools  to  fully  direct  program  goals  and  efforts 
from  Day  1.  (Olson,  1999) 

To  further  exemplify  these  concepts,  SMART  has  been  defined  as  an 
initiative  that  promotes  the  robust  use  of  M&S  integrated  across  acquisition 
programs  and  phases  to  reduce  Total  Ownership  Cost  (TOC),  provide  quicker 
delivery  of  products  to  the  field,  while  simultaneously  increasing  military  utility 
and  worth  (SMART  2001). 

The  Planning  Guidelines  for  SMART  state  that  a  simulation  support  plan 
(SSP)  is  needed  to  support  the  use  of  M&S  in  programs.  The  SSP  is  designed  to 
provide  a  clear  direction  how  the  application  of  M&S  tools  support  the 
development  of  the  system.  The  Guidelines  further  state  that  the  SSP  defines 
the  timing  of  M&S  tools  during  concept  exploration  and  system  development 
(Ellis,  Kern,  and  Hollis,  2000). 

The  Guidelines  also  address  the  issue  of  developing  Modeling  and 
Simulations  through  SMART.  More  specifically,  the  Guidelines  define  how  M&S 
can  be  used  conceptually,  through  the  application  of  SMART,  to  aid  in  the 
functions  of  combat  development,  design  and  engineering,  logistics  and  support, 
test  and  evaluation,  and  training.  (Ellis,  Kern,  and  Hollis,  2000)  Two  of  these  will 
be  described  below: 

Combat  Development.  The  Guidelines  state  that  an  Integrated  Concept 
Team  (ICT)  should  develop  the  SSP.  To  effectively  develop  the  SSP,  the  ICT 
should  use  M&S  in  the  following  ways: 

•  The  ICT  can  look  at  the  feasibility  of  reuse  of  M&S  throughout 
the  life  cycle. 

•  The  ICT  should  examine  other  current  Army,  OSD,  and  sister 
Services’  programs. 

•  Upgrade  existing  M&S  while  verifying  and  validating  the  M&S 
for  its  new  use. 
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•  Developing  new  M&S.  (Ellis,  Kern,  and  Hollis,  2000) 

Design  and  Engineering.  The  intent  of  SMART  and  M&S  in  the  design 

or  engineering  process  is  to  reduce  costs  by  using  one  centralized  database. 

More  specifically,  the  guidelines  state: 

M&S  used  as  part  of  the  collaborative  environment  (CE)  by  ICTs  or 
IPTs  should  be  able  to  support  detailed  design  and  reduce  design 
risk  by  allowing  all  functional  disciplines  to  work  from  the  same 
design  database.  A  reduced  number  of  engineering  change 
proposals  (ECPs)  is  an  important  result  of  this  CE.  Employment  of 
hardware-  or  software-in-the-loop  (HW/SWIL)  M&S  is  also  part  of 
this  CE  and  results  in  significant  risk  reduction  in  test  and 
evaluation  through  planning,  hardware  checkout,  and  mission 
rehearsal.  Finally,  this  concept  of  a  SMART  CE  allows  transition  to 
production  to  take  place  with  reduced  risk  by  the  electronic  transfer 
of  digital  design  data  directly  to  the  manufacturing  floor.  (Ellis,  Kern, 

Hollis,  2000). 


With  the  previous  two  functional  areas,  it  has  been  demonstrated  that 
SMART,  through  M&S,  provides  the  tools  to  reduce  or  affect  life  cycle  costs. 
Price  identifies  five  Army  models  that  provide  that  type  of  life  cycle  cost 
management.  These  models  work  through  improving  logistics  while  meeting 
SMART  readiness  goals.  These  systems  are:  ASOAR,  SESAME,  COMPASS, 
ACEIT,  and  LCET.  A  table  is  provided  below  that  defines  whether  or  not  these 


models  can  be  used  to  address  either  readiness  or  cost  issues. 


Acronym 

Model  Name 

Type 

ASOAR 

Achieving  a  System  Operational  Availability 
Requirement 

Readiness 

SESAME 

Selected  Essential-Item  Stock  for  Availability 
Method 

Readiness 

COMPASS 

Computerized  Optimization  Model  for 

Predicting  and  Analyzing  Support  Structures 

Readiness  and 
Cost 

ACEIT 

Automated  Cost  Estimating  Integrated  Tools 

Cost 

LCET 

Logistics  Cost  Estimating  Tool 

Cost 

i 

(From  Price,  2002) 

Table  1. 

Linked  /  Integrated  Army  Models 
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C.  DEPARTMENT  OF  THE  NAVY 

The  US  Navy  Modeling  and  Simulation  Management  Office  (NAVMSMO) 
is  the  lead  agency  for  Modeling  and  Simulation  for  the  Navy.  As  such,  the  office 
is  charged  with  managing  modeling  and  simulation  issues  including  maintaining 
and  disseminating  documentation.  An  Instruction  from  the  Secretary  of  the  Navy, 
which  is  the  defining  document  for  Modeling  and  Simulation,  can  be  found  at  the 
NAVMSMO  website.  This  document  establishes  the  Navy’s  definition  of 
modeling  and  simulation  as  follows: 

Models  include  any  physical,  mathematical,  or  otherwise  logical, 

representation  of  a  system,  entity,  phenomenon  or  process. 

Simulations  include  a  method  of  implementing  a  model  over  time. 

(SECNAV  5200.38A) 

The  definition  that  the  Navy  applies  to  these  terms  is  comprehensive  in 
nature.  Furthermore,  the  definition  lends  itself  to  SBA,  which  was  earlier 
established  as  an  iterative,  integrated  process  that  incorporates  modeling  and 
simulation  to  achieve  life  cycle  cost  savings  and  reduction  of  risk. 

The  Navy  has  established  objectives  for  modeling  and  simulation  in  order 
that  M&S  is  treated  uniformly  throughout  the  Service.  These  objectives  are: 

•  Modeling  and  simulation  and  associated  information  technology 
will  be  applied  consistently  across  each  of  the  four  pillars  of 
naval  M&S.  These  pillars  are:  operations  and  experimentation; 
training;  acquisition;  and  analysis  and  assessment. 

•  Modeling  and  simulation  technology  shall  be  readily  available  to 
the  naval  warfighter. 

•  Modeling  and  simulation,  and  its  underlying  data,  will  be 
consistently  applied  across  the  Navy-Marine  Corps  team  afloat 
or  ashore,  at  home  or  deployed. 

•  Investment  in  modeling  and  simulation  technology  will  be  cost 
effective,  have  measurable  benefits,  and  build  on  the 
Department  of  Defense  (DoD)  and  commercial  capabilities  and 
standards. 

•  The  DON  Science  and  Technology  (S&T)  efforts  will  ensure  the 
development  of  technologies  to  meet  the  modeling  and 
simulation  needs  of  the  Sailor,  the  Marine,  and  the  DON. 
(SECNAV  5200.38A) 
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The  Instruction  goes  on  to  state  that  M&S  applications  shall  be 
interoperable  throughout  all  levels.  To  support  the  intent  of  interoperability  as 
well  as  the  objectives  listed  above,  the  Navy  developed  a  Modeling  and 
Simulation  Master  Plan. 

The  Modeling  and  Simulation  Master  Plan  describes  the  Navy’s  intent  to 
implement  M&S  throughout  its  programs.  The  plan  emphasizes  three  general 
areas  that  affect  each  program.  These  are  Assessment,  Training,  and  Analysis. 
The  Navy’s  objectives,  as  stated  in  the  guide  are  to  use  M&S  to  simulate  before 
significant  action  is  taken,  integrate  models  of  all  types  to  support  training,  and 
allow  collaborative  mission  planning  while  combined  with  M&S,  to  support 
planning  and  operations.  (NAVMSMO,  1997) 

Assessment  is  defined  as  making  better  and  more  informed  decisions. 
Training  includes  those  efforts  that  will  improve  warfighter  skills  and  capacity. 
Acquisition  is  focused  on  reduced  cycle  time,  reduced  cost,  and  optimized 
systems.  (NAVMSMO,  1997) 

To  support  these  general  areas,  the  Navy  has  identified  building  blocks  as 
Model  Standards,  Data  Standards,  and  Communications  (NAVMSMO,  1997). 
Figure  3  depicts  this  relationship.  It  should  be  noted  that  the  general  areas  of 
Assessment,  Training,  and  Acquisition  are  mutually  coexistent  and  not  meant  as 
a  flow  upward  into  the  next  level. 
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Figure  3. 

M&S  Building  Blocks  Supporting  the  Areas  of  Assessment,  Training,  and 

Acquisition 

The  Modeling  and  Simulation  Master  Plan  uses  the  above  representation 
to  reflect  the  utilization  of  the  Synthetic  Battlespace.  The  concept  of  this 
battlespace  is: 

...the  collection  of  models  and  simulations  and  associated 
databases;  the  networks,  including  the  supporting  connectivity, 
interfaces,  associated  hardware,  and  underlying  software  and 
protocols  and  the  real  world  operators,  ranges,  platforms,  systems, 
and  hardware  (NAVMSMO,  1997). 

The  plan  goes  on  to  state  that  the  intent  is  to  use  these  elements  on  an  as 
needed  basis  rather  than  simultaneously.  These  elements  would  be  used  to 
represent  operations,  activities,  or  processes,  at  a  level  acceptable  to  the  user 
(NAVMSMO,  1997).  So  the  user,  in  the  Navy,  may  approach  program  issues 
with  a  variety  of  resources.  Such  resources  can  be  used  to  model  oceanic 
conditions,  three-dimensional  design  capabilities,  or  future  training  requirements. 
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As  the  system  matures,  the  M&S  uses  are  refined  and  updated  in  the 
battlespace.  The  need  for  interoperability  must  be  maintained  and  addressed  in 
each  model  or  simulation  developed. 

The  building  blocks  of  the  synthetic  battlespace  were  defined  as  Modeling 
Standards,  Data  Standards,  and  Communications.  These  elements  are  more 
explicit  as  described  below: 

•  The  Modeling  Standards  building  block  encompasses  the 
development  and  promulgation  of  standards,  the  process  to 
ensure  models  realistically  represent  valid  Naval  doctrine  and 
behavior,  the  establishment  of  a  M&S  Resource  Repository 
(MSRR),  the  “standardized”  models  themselves,  and  the 
designation  of  executive  agents  for  specific  areas  and 
proponents  for  configuration  management  of  specific  models  or 
programs. 

•  The  Data  Standards  building  block  encompasses  the  process  to 
ensure  comprehensive,  consistent  and  authoritative  data  are 
available  to  support  the  synthetic  battlespace,  the  development 
and  enforcement  of  data  standards,  the  actual  filling  of  the 
databases,  and  access  to  the  data  by  the  users  through 
DMSO’s  MSRR. 

•  The  Communications  building  block  encompasses  the 
development  of  evolving  communication  standards,  formats, 
and  protocols  necessary  for  connectivity  and  data  exchange 
over  a  network;  the  long-range  plan  for  building  the  physical 
communication  nodes  and  acquiring  the  hardware;  the  protocols 
enabling  federations  of  models  to  interact;  and  multi-level 
security  hardware  and  standards  necessary  for  classified 
operations.  (NAVMSMO,  1997) 


The  ideals  of  M&S,  found  in  the  Naval  policies  and  Master  Plan,  can  also 
be  found  in  the  Air  Force  policies  and  plans. 


D.  DEPARTMENT  OF  THE  AIR  FORCE 

The  US  Air  Force  (USAF)  also  has  provided  direction  to  its  Service 

regarding  the  use  of  M&S  in  its  programs.  USAF  Policy  Directive  16-10, 

modeling  and  Simulation  (M&S)  Management,  clarifies  USAF  policies  as: 

•  Provide  M&S  management  policies  and  procedures  which 
identify  opportunities  and  implement  practical  programs  to 
reduce  costs;  eliminate  duplication;  promote  exportability  of 
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models  and  simulations;  promulgate  standards;  streamline 
model  development;  and  create  seamless,  interoperable,  live, 
virtual,  and  constructive  distributed  simulations  responsive  to 
the  users’  needs. 

•  Where  appropriate,  promote  M&S  investments  which 

incorporate  enhancements  of  Department  of  Defense  (DoD) 

M&S  technologies  in  support  of  operational  needs  and  the 
acquisition  process. 

•  Promote  the  use  of  common  tools,  methodologies,  and 
databases  to  the  maximum  extent  possible. 

•  Establish  standards  and  protocols  when  needed  or  implement 
existing  standards  and  protocols  when  available  that  promote 
the  internetting,  data  exchange,  open  system  architecture,  and 
software  reusability  of  M&S  applications. 

•  Publish  a  M&S  Master  Plan  and  a  M&S  Investment  Plan. 

(USAF,  16-10) 

The  goals  of  the  Air  Force,  with  respect  to  Modeling  and  Simulation,  are 
similar  to  those  of  the  Navy  and  Army.  All  Services  encourage  reusability,  use 
across  systems  and  functions,  and  implementation  of  DoD  overarching  policy. 
The  Air  Force  policy  addresses  the  logistics  area  as  defined  in  the  Charter  for  the 
Air  Force  Logistics  Modeling  and  Simulation  IPT.  In  this  charter,  the  Air  Force’s 
vision  is: 

To  develop  Air  Force  Logistics  Modeling  &  Simulation  (M&S)  in 
support  of  Air  Force  and  Joint  exercises,  wargames, 
experimentation,  studies,  analyses,  and  acquisitions.  This  support 
will  span  the  three  M&S  functional  areas:  Training,  Analysis,  and 
Acquisition.  (AF  IPT,  2001) 

The  master  plan  developed  for  the  Air  Force  regarding  Modeling  and 
Simulation  sets  forth  the  intentions  and  policies  regarding  the  use  of  M&S  in  Air 
Force  programs.  The  Executive  Summary  states  that  the  goal  of  M&S  is  to 
develop  a  capability  using  interoperable  M&S  systems  to  provide  warfighters  and 
decision-makers  the  tools  to  ensure  readiness  across  the  full  spectrum  of  conflict 
(HQ  USAF,  1995).  The  plan  goes  on  to  define  the  vision  of  M&S  to  develop  a 
Joint  Synthetic  Battlespace  supporting  better  decisions  and  warfighting  skills  to 
build  the  world’s  most  respected  air  and  space  force  for  the  Joint  Force 
Commander  (HQ  USAF,  1995).  It  is  the  intent  of  the  Air  Force  to  have  a 
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battlespace  representation  for  all  elements  in  the  acquisition  cycle  regardless  of 
position  within  the  acquisition  cycle.  To  achieve  this  vision,  the  Air  Force  has 
defined  the  Joint  Synthetic  Battlespace  as  a  vehicle  for  development  of  future 
forces,  concepts,  systems,  and  doctrine  in  a  simulated  environment  where  Air 
Force  roles  and  missions  will  be  appropriately  and  accurately  represented  (HQ 
USAF,  1995). 

The  Air  Force  intends  to  use  modeling  and  simulation  to  accomplish  and 
meet  three  functions  within  the  Service.  The  first  function  is  to  provide  improved 
quality  management  for  the  systems  or  programs  throughout  the  Air  Force. 
Second,  M&S  will  be  used  to  improve  the  level  of  training  used  by  Air  Force 
personnel.  Finally,  the  Air  Force  will  improve  the  structure  of  programs,  including 
upgrading  existing  standards  and  protocols.  (HQ  USAF,  1995) 

The  Air  Force  also  has  identified  two  separate  processes  in  which  M&S 
will  be  applied  throughout  the  Service.  These  processes  are  Analysis  and 
Training.  These  two  elements  are  also  a  part  of  the  DON  structure  discussed 
previously.  Analysis  is  aimed  at  making  better  decisions,  while  training  is 
targeted  at  developing  better  skills  (HQ  USAF,  1995).  To  this  end,  the  Air  Force 
has  also  developed  a  pyramid  similar  to  that  of  the  Army’s  in  which  functions  flow 
upward  into  more  advanced  or  general  uses.  At  the  same  time,  the  Analysis  and 
Training  elements  are  considered  the  building  blocks  for  this  pyramid.  While 
DON  uses  Analysis  and  Training  (and  Acquisition)  as  functional  groups,  the  Air 
Force  has  identified  these  as  supporting  the  upward  flowing  elements.  This  can 
be  seen  in  Figure  4  below. 
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Figure  4. 

Modeling  and  Simulation  Hierarchy 


The  Modeling  and  Simulation  Master  Plan  further  develops  the  intent  of 
this  hierarchal  pyramid  as  follows: 

•  System/Sub-system/Component  (Engineering)  -  Design, 
cost,  manufacturing,  and  supportability.  Provides  measure  of 
performance. 

•  Engagement/Sub-mission  -  Evaluation  of  system 
effectiveness  against  enemy  systems.  Provides  measures  of 
effectiveness  at  the  system  level. 

•  Mission  (Battle)  -  Effectiveness  of  a  force  package  or  multiple 
platforms  performing  specific  mission. 

•  Campaign  (Theater)  -  Outcomes  of  joint/combined  forces  in 
campaign/theater  level  conflict.  Provides  measures  of  outcome 
and  other  high  level  measures  of  merit.  (HQ  USAF,  1995) 
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In  review  of  this  chapter,  the  paper  has  covered  the  basic  guidance 
from  DoD  and  its  Components  as  follows 

•  DoD:  DoDD  5000.59,  DoDI  5000.61,  formation  of  EXCIMS,  and 
STEP  guidelines. 

•  Army:  AR  5-1 1  and  SMART. 

•  Navy:  SECNAV  5200. 38A  and  Navy  Modeling  and  Simulation 
Master  Plan. 

•  Air  Force:  Air  Force  Policy  Directive  16-10  and  US  Air  Force 
Modeling  and  Simulation  Master  Plan. 

Throughout  these  documents,  the  intent  of  providing  guidance  on 
modeling  and  simulation  has  remained  constant.  Each  of  the  Services 
has  identified  how  Modeling  and  Simulation  will  be  treated  within  the 
Service.  Typically,  this  means  creating  a  database  of  models,  simulations, 
representations,  and  environments  that  accurately  reflect  the  Service’s 
mission.  The  plans  go  on  to  state  how  the  use  of  M&S  will  flow  through 
the  programs  and  systems  to  achieve  the  end  vision  of  theater 
involvement.  Ultimately,  it  is  up  to  the  program  manager  to  follow  these 
guidelines  in  order  to  leverage  the  existing  models/simulations  against  the 
development  of  the  program. 
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IV.  DATA  GATHERING  AND  RESULTS 


Field  Data  is  invaluable  in  terms  of  addressing  the  transitioning  of  the  SBA 
concept  to  real  time  events.  This  study  undertook  a  survey  to  gather  data  that 
described  the  uses  of  SBA  in  real  programs.  The  question  then  was  to  identify 
which  programs  and  what  Service  would  be  appropriate  in  terms  of  gathering 
data  to  study.  Once  the  potential  target  sample  had  been  established,  a  survey 
could  be  sent  to  the  appropriate  personnel  for  completion  and  return. 

A.  SURVEY  STRUCTURE 

In  designing  any  survey,  one  must  determine  the  fundamental  question 
and  any  underlying  sub-questions  that  need  to  be  answered.  Done  well,  a 
survey  will  obtain  critical  data  from  which  meaningful  conclusions  and  insights 
may  be  drawn. 

In  order  to  determine  if  there  are  economic  breakpoints  in  programs,  the 
survey  established  the  fundamental  question  of  the  survey  as  “What  is  the 
degree  of  use  of  SBA  in  military  programs,  given  the  pre-existing  direction 
promulgated  by  DoD?”  There  is  a  significant  amount  of  information  on  SBA  that 
has  been  disseminated  from  the  highest  levels.  Each  Service  has  established  its 
component  lead  for  Modeling  and  Simulation.  Finally,  the  amount  of  information 
available  on  the  Internet  is  staggering  in  quantity  and  quality.  Given  the  amount 
of  information  on  SBA  and  its  concepts,  one  would  expect  that  well-informed 
PMs  will  incorporate  these  concepts  into  their  programs. 

Therefore,  the  real  question  likely  should  be  “How  has  one  Service 
derived  benefits  from  SBA  in  their  programs?”  However,  one  might  also  inquire, 
as  an  underlying  question,  as  to  the  ways  in  which  SBA  is  utilized  in  the  program. 
This  is  conceptually  different  than  the  fundamental  question  set  forth  above.  In 
the  fundamental  question,  one  is  trying  to  determine  how  the  use  of  SBA  in 
programs  has  been  put  forth  (e.g.,  phases  and  magnitude).  In  the  second 
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question,  one  would  search  for  the  ways  in  which  SBA  is  present  in  the 
programs. 

A  second  underlying  question  might  be  stated  as  to  why  the  funding  for 
SBA  is  typically  done  in  the  initial  phases.  One  would  assume  that  early  funding 
of  SBA  in  programs  would  provide  more  meaningful  results  than  incorporation 
later  in  the  life  of  the  program.  The  intent  of  the  survey  is  to  assess  the  validity  of 
this  assertion. 

The  final  underlying  question  addresses  the  kinds  of  benefits  derived  from 
the  use  of  SBA.  A  program  manager  must  believe  that  there  is  an  inherent 
recompense  for  using  SBA  in  their  program.  If  not,  then  there  would  be  little 
reason  to  pursue  the  use  of  SBA  aside  from  meeting  higher  headquarters' 
requirements.  The  survey  addressed  the  question  of  what  types  of  benefits  exist 
and  how  rewarding  will  those  benefits  were  for  the  program  overall. 

B.  WHO  TO  SURVEY 

The  need  for  a  meaningful  sample  of  a  population  is  crucial  to  an  effective 
survey.  Typically,  a  survey  will  result  in  a  return  rate  of  approximately  one  to  two 
percent.  A  higher  rate  of  completion  can  be  achieved  through  the  notation  of 
vested  interest.  In  other  words,  should  the  individual  receiving  the  survey  believe 
that  the  survey  is  meaningful  in  some  way,  that  individual  would  be  more  inclined 
to  complete  and  return  the  survey. 

In  this  study,  the  Department  of  the  Navy  (DoN)  was  identified  as  a 
potential  source  for  data  points.  The  reasoning  for  this  selection  is  that  DoN  has 
a  wide  variety  of  military  programs,  these  programs  represent  all  levels  of 
Acquisition  Categories  (ACAT),  and  a  listing  of  these  programs  was  readily 
accessible. 

C.  ISSUES  OF  THE  SURVEY 

It  can  be  safely  stated  that  most  surveys  are  not  without  their  weak  points. 
The  survey  used  in  this  study  was  intended  to  derive  initial  feedback  as  to  the 
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use  of  SBA  in  military  programs.  It  was  not,  however,  designed  to  be 
comprehensive. 

One  weak  area  in  the  survey  was  its  inability  to  confine  the  respondents 
into  a  uniform  response.  Of  the  surveys  received,  six  (6)  could  not  be  placed  in 
the  quantifiable  set  as  the  responses  were  narrative  rather  than  measurable. 
This  was  not  entirely  without  benefit  as  the  study  was  able  to  use  the  narrative 
information. 

A  second  area  of  concern  was  that  the  respondents  did  not  have  the 
information  that  was  required  by  the  survey.  This  might  be  a  function  of  the 
program  being  compartmentalized  or  a  respondent  being  out  of  the  information 
loop.  Another  aspect  of  this  concern  is  that  the  program  manager  might  be 
disinclined  to  release  information  that  is  considered  sensitive.  Additionally,  the 
respondent  sometimes  stated  that  he  did  not  know  or  the  response  indicated  that 
the  respondent  did  not  understand  the  question. 

A  third  concern  is  that  the  survey  did  not  make  allowances  for  the  ranges 
given  in  the  responses.  For  example,  the  survey  asked  for  the  phase  in  which 
M&S  funding  was  used.  Responses  oftentimes  indicated  more  than  one  phase 
in  which  M&S  funding  had  been  used. 

Finally,  the  survey  did  not  allow  for  the  size  differential  between  programs. 
The  survey  accessed  all  ACAT  levels.  However,  the  impact  that  a  multi-billion 
dollar  program  would  have  on  the  survey  is  greatly  different  than  a  much  smaller 
million-dollar  program. 

D.  OUTPUT  FROM  THE  SURVEY 

1.  Responses 

There  were  471  total  programs  listed  on  the  DoN  ACAT  Matrix  that  were 
under  the  cognizance  of  132  separate  program  managers.  Of  these  471 
programs,  416  surveys  were  emailed  to  the  cognizant  PM  (107  total).  There 
were  several  programs  that  did  not  list  an  email  address  or  the  survey  was 
returned  as  undeliverable.  The  total  number  of  surveys  sent  out  represents 
88.32%  of  the  total  number  of  programs  possible.  Of  the  surveys  sent  out,  there 
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were  50  responses  that  were  completed  and  4  responses  that  did  not  offer 
financial  data,  but  did  offer  informative  perspectives  on  the  use  of  SBA  in  their 
program.  Fifty-four  responses  provided  a  12.98%  response  rate  (of  the  416 
surveys  sent),  significantly  higher  than  typical  survey  response  rates. 

2.  Assumptions  and  Generalizations 

In  assessing  the  responses  to  the  survey,  it  was  necessary  to  make 
assumptions  and  generalizations.  One  such  assumption  occurred  when  a 
response  was  provided  that  had  a  range  (e.g.,  Program  phase  was  partly  in  EMD 
and  partly  in  Production).  In  these  cases,  the  range  was  averaged  and  recorded. 
However,  if  a  program  indicated  that  it  would  use  M&S  in  more  than  one  phase, 
the  more  recent  phase  was  recorded.  The  reasoning  was  that  programs  would 
most  likely  have  M&S  be  more  significant  in  the  EMD  phase  rather  than  PDRR. 
These  were  the  two  most  frequent  choices  albeit  only  a  small  number  of  surveys 
required  this  manipulation.  Additionally,  in  terms  of  either  Life  Cycle  Cost  (LCC) 
avoidance  or  factored  in  LCC  avoidance,  a  recorded  response  of  zero  means 
that  that  either  there  were  no  preplanned  LCC  avoidances  or  that  no  quantifiable 
amount  could  be  discerned  from  the  response.  Another  convention  is  that  some 
responses  indicated  that  the  program  was  in  Post  Production.  In  these  cases, 
the  recorded  phase  was  Production.  The  program  amount  listed  on  the  survey 
was  another  area  in  which  a  convention  needed  to  be  derived.  If  a  program’s 
value  was  more  than  $5  billion  dollars,  the  survey  needed  to  have  all  financial 
information  completed  in  order  to  be  recorded.  Otherwise,  the  survey  would 
contain  significant  outliers  in  the  compilation  of  the  data.  This  also  implies, 
correctly,  that  some  surveys  were  recorded  that  were  incomplete.  In  doing  so, 
the  data  would  provide  the  most  basic  of  information,  rather  than  a  purely 
empirical  study.  Finally,  the  survey  did  not  baseline  the  funding  dollars  to  a 
specific  fiscal  year. 
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3.  Preliminary  Data 

The  surveys  returned  diverse  information  and  data  from  all  levels  of 
acquisition  categories.  Of  significance  is  that  eighteen  respondents  funded  SBA, 
but  slightly  more  than  half  of  those  did  not  recognize  cost  avoidance  in  their 
programs.  Of  the  seven  that  did  achieve  cost  avoidance,  five  of  them  received 
more  in  savings  than  expended  on  SBA.  One  may  conclude  that  respondents 
were  not  able  to  accurately  judge  the  downstream  effects  of  the  investment  or 
that  the  program  had  not  progressed  enough  to  establish  actual  savings. 

The  other  significant  observation  is  that  the  ACAT  Level  III  programs  (who 
responded)  are  providing  the  most  percentage  in  terms  of  SBA  funding.  It  may 
be  concluded  that  the  reasoning  for  this  is  the  acquisition  cost  of  a  ACAT  Level  III 
program  is  significantly  less  than  that  for  ACAT  Level  I.  Therefore,  although  an 
ACAT  Level  I  program  might  spend  more  on  SBA,  the  overall  percentage  is 
significantly  less  than  lesser  programs. 
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Cost 

Avoidance  as 
%of 


SBA  as  %  of 

Acquisition 

ACAT  Level 

Acquisition  Cost 

Cost 

1 

0.70 

2.60 

2 

0.20 

0.00 

2 

2.60 

1.70 

3 

1.00 

1.50 

3 

1.10 

0.00 

3 

1.40 

0.00 

3 

2.00 

2.60 

3 

2.90 

0.00 

3 

2.90 

0.00 

3 

4.40 

0.00 

3 

8.10 

0.00 

3 

11.30 

0.00 

3 

11.30 

0.00 

3 

19.80 

0.00 

4 

0.10 

0.20 

4 

1.00 

1.90 

4 

1.10 

0.00 

4 

4.60 

0.90 

Tab  le  2 

Comparison  of  Projected  SBA  Cost  to  Anticipated  Cost  Avoidance 

4.  Initial  Insights 

There  were  numerous 

insights  that  were 

derived  from  the  surveys 

returned.  The  first  view  was 

that  there  was  little 

or  no  M&S  sharing  among 

programs.  There  were  numerous  examples  provided  that  stated  the  program 
used  M&S  to  simulate  platforms  or  testing.  It  could  be  inferred  that  the  most 
programs  needed  to  develop  program  specific  simulations  rather  than  using  a 
generic  software  device  for  most  programs.  A  second  observation  is  that  most  of 
the  M&S  funding  has  already  occurred  and  this  is  consistent  with  the  point  that 
most  programs  are  in  the  Production  Phase.  A  correlated  observation  is  made 
that  87.8%  of  SBA  funding  is  typically  made  with  RDT&E  funds.  Those 
responding  to  questions  regarding  cost  avoidance  were  able  to  identify  a  life 

cycle  savings.  The  $3.8  million  dollars  in  life  cycle  cost  avoidance  identified  were 
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not  pre-planned.  In  fact,  in  only  one  case  were  costs  of  $50  million  dollars 
planned  to  be  avoided  through  the  use  of  SBA. 

5.  Significant  Comments  from  Respondents 

A  significant  number  of  respondents  provided  extremely  useful  and 
intuitive  comments  when  completing  the  survey.  These  comments  are  provided 
below: 


a.  Computer  Modeling 

In  one  program,  Computer  Modeling  has  been  used  to  analyze,  in 
high  resolution,  structural  composition.  The  intent  of  the  analysis  was  specifically 
to  reduce  ground  and  flight  testing.  An  interior  simulation  was  used  to  design 
controls  within  the  system.  The  program  went  on  to  study  vulnerability  issues 
and  modeling  through  the  use  of  extensive  computer  tools.  The  program 
incorporated  SBA  through  the  use  of  vulnerability  analyses,  susceptibility  studies, 
structural  analysis,  and  design  functions.  The  benefit  derived  from  the  uses  of 
SBA  were  cost  avoidance  by  eliminating  developmental  testing  and  design  as  a 
function  of  simulations. 

b.  CAD/CAM 

Another  program,  currently  in  EMD  phase,  estimates  that  between 
1%  -  2%  of  their  total  budget  line  will  be  spent  on  M&S  throughout  the  life  of  the 
program.  This  is  consistent  with  the  earlier  observation  of  the  same  nature  in  the 
Preliminary  Data  section.  The  program  plans  on  using  such  tools  as  Computer 
Aided  Design  (CAD),  Computer  Aided  Manufacturing  (CAM),  finite  element 
analysis,  and  survivability  studies.  A  result  of  these  tools  and  actions  in  the  past 
is  the  cost  avoidance  realized  in  the  program.  The  computer  models  were  found 
in  existence  already,  rather  than  having  modeling  software  exclusively  made  for 
the  program.  Furthermore,  the  program  personnel  proactively  worked  with  the 
prime  contractor  to  find  M&S  tools  already  in  use  in  other  military  programs. 
Such  an  emphasis  was  placed  on  SBA  during  PDRR,  that  the  program  specified 
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a  lead  team  member  to  coordinate  M&S  functions.  Finally,  the  program 
recognized  the  leadership  direction  of  DoD  and  DMSO  in  providing  commonality 
to  military  programs. 


c.  Virtual  Prototyping 

A  different  program  has  utilized  virtual  prototyping  in  the  initial 
phase  of  the  program.  In  addition,  the  program  plans  on  using  M&S  to  derive  a 
reduction  in  engineering  changes  and  product  rework.  In  using  M&S,  the  team 
believes  that  training  and  T&E  will  experience  efficiencies,  thereby  reducing  cost. 
Finally,  this  team  developed  a  database  that  allowed  an  easy  access  to 
information  by  both  the  Government  team  and  the  prime  contractor.  The 
respondent  also  stated  that  use  of  SBA  was  a  significant  resource  for  the  design 
teams.  The  database  offered  information  for  use  in  risk  management,  product 
model  development,  and  requirements. 

d.  Production  Data 

One  program  plans  on  using  SBA  funding  to  update  and  maintain 
the  production  phase  of  the  program.  This  program  implemented  high  fidelity 
simulations  to  model  the  hardware  in  use.  By  doing  so,  the  program  eliminated 
the  cost  of  multiple  testing  and  functioning  of  the  item.  The  program  went  on  to 
say  that  SBA  is  a  way  of  life  in  this  type  of  program.  Once  the  high  fidelity 
modeling  software  has  been  developed,  it  is  used  throughout  development, 
production,  and  product  improvement. 

e.  Cost  Savings 

Modeling  and  Simulation  have  been  mandated  in  another  program 
to  achieve  cost  savings.  These  savings  are  a  result  of  design/concept  trade-offs, 
lifecycle  tests,  system  integration,  and  transitioning  the  program  from 
development  to  deployment.  By  the  mandated  use  of  SBA,  the  program  has 
been  able  to  field  the  most  effective  and  affordable  weapon. 
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E.  INFERENCES  DRAWN  FROM  THE  DATA 

Earlier,  this  chapter  established  the  question  of  “What  is  the  degree  of  use 
of  SBA  in  military  programs,  given  the  pre-existing  direction  promulgated  by 
DoD?”  In  attempting  to  answer  the  question,  this  study  surveyed  a 
comprehensive  listing  of  Navy  programs.  Limited  conclusions  from  the  data  are: 

•  Cost  avoidance  exists  for  those  programs  willing  to  seek  it.  However, 
in  this  study,  only  one  program  sought  to  incorporate  specific  savings. 

•  Sharing  of  cost  avoidance  methods  was  not  present  in  a  majority  of 
those  programs  responding  to  the  survey. 

•  SBA  was  found  to  be  generally  disseminated  through  training,  although 
some  programs  had  yet  to  learn  of  SBA  and  its  concepts.  Many 
programs  already  are  doing  M&S  and  not  realizing  that  this  meets  the 
criteria  of  SBA. 

•  Appropriately,  when  SBA  is  incorporated  into  the  program,  the  funding 
is  early  in  the  program.  However,  only  1%  of  the  average  Program 
Acquisition  Cost  is  spent  on  SBA  methods. 

•  The  benefits  derived  from  the  use  of  SBA  were  found  to  be  directly 
related  to  the  program  in  the  form  of  requirements  development, 
platform  qualification,  and  product  testing. 

•  The  benefits  of  current  and  future  design  changes,  risk  reduction,  and 
M&S  interoperability  were  found  to  be  derived  almost  entirely  in  the 
PDRR  and  EMD  Phases  of  the  program.  This  is  consistent  with  the 
earlier  observation  that  most  funding  is  of  RDT&E  in  nature. 

F.  DISCUSSION  OF  PRIMARY  THESIS  QUESTION 

The  intent  of  this  study  was  to  determine  if  there  are  specific  times  in  the 
acquisition  life  cycle  in  which  the  program  manager  can  use  SBA  to  provide 
effectively  cost  avoidance  in  the  system.  The  comparison  of  the  old  series  DoD 
5000  acquisition  life  cycle  and  the  revised  life  cycle  is  shown  below.  The  upper 
scale  represents  the  new  acquisition  life  cycle  as  set  forth  in  a  memorandum 
from  the  Honorable  Paul  Wolfowitz  (Undersecretary  of  Defense)  dated  30  Oct  02. 
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The  lower  scale  represents  the  traditional  timeline  for  the  acquisition  life  cycle  set 
forth  in  DoD  series  5000. 
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The  survey  has  shown  that  there  are  times  in  which  the  use  of  Simulation 
Based  Acquisition  is  appropriate.  It  should  be  noted,  however,  that  different 
applications  of  SBA  occur  at  different  time  in  the  program. 

For  programs  entering  Concept  Exploration,  SBA  methods  should  be  fully 
built  into  the  system  as  it  develops.  For  example,  the  use  of  virtual  prototyping, 
CAD/CAM,  and  shared  databases  are  extremely  effective  in  this  phase.  The 
program  manager  has  the  opportunity  to  significantly  affect  downstream  issues 
and  control  costs  for  the  future. 

When  the  program  enters  System  Development  and  Demonstration,  other 
SBA  methods  would  complement  those  from  the  first  phase.  The  program 
manager  would  want  to  add  computer  modeling  for  form,  fit,  and  function, 
reconfigurable  prototypes  to  mirror  design  changes  in  real  time,  and  consider 
logistical  issues. 
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In  Production  and  Development,  the  system  design  has  already  been 
solidified.  Therefore,  the  program  manager  will  want  to  apply,  in  addition  to 
those  SBA  actions  already  taken,  reusable  simulations  for  training  and  live  fire 
exercises,  interoperability  models  for  follow  on  efforts  or  other  Service  programs, 
and  application  of  the  system  in  force  or  mission  level  distributive  interactive 
simulations  (DIS).  This  final  action  relates  to  the  hierarchy  of  M&S  that  was 
discussed  under  the  chapter  on  Guidance. 

Therefore,  it  can  be  seen  that  at  each  stage  in  the  acquisition  life  cycle,  it 
is  economically  practical  to  employ  SBA  in  almost  every  program  type  and  ACAT 
level.  Those  responding  to  the  survey  indicated  that  savings  were  achieved  in 
each  phase.  Finally,  it  should  be  noted  that  the  more  that  SBA  is  funded  in  the 
initial  stages  of  the  program,  the  more  effective  it  is  as  the  program  progresses. 
Aside  from  monetary  savings,  risk  reduction  was  a  significant  issue  as  reported 
by  survey  returns. 
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V.  CASE  STUDIES 


Simulation  Based  Acquisition  has  been  used  or  is  being  used  in  military 
programs  since  implementation  of  the  guidance.  In  a  statement  to  the  Senate 
Committee  on  Armed  Services,  Dr.  Kenneth  Oscar  stated: 

Through  Simulation  Based  Acquisition  (SBA),  the  Army  is  using 
modeling  and  simulation  (M&S)  to  reduced  acquisition  costs,  total 
ownership  costs,  and  time  to  initial  operating  capability,  while 
increasing  the  military  utility  and  quality  of  a  fielded  system 
throughout  its  lifecycle  (Oscar,  1998). 

Additionally,  the  US  Army  believes  and  supports  the  implementation  of 

SBA... 

...across  acquisition  programs  and  phases  to  reduce  total 
ownership  costs,  provide  quicker  delivery  of  products  to  the  field, 
while  simultaneously  increasing  military  utility  and  worth  (Truelove, 

1999). 


In  concert  with  his  pronouncements,  above,  Truelove  stated  that  the  Army 
had  identified  four  flagship  programs  in  which  Simulation  Based  Acquisition  is 
being  used.  These  were  the  Future  Scout  and  Cavalry  System  (FSCS),  the 
Crusader,  the  AH  -  64  Apache  Longbow  Helicopter,  and  the  UH  -  60  Blackhawk 
helicopter.  The  use  of  SBA  in  these  programs  provides  excellent  insight  into 
the  application  techniques  as  well  as  the  benefits  derived. 
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A.  FUTURE  SCOUT  CAVALRY  SYSTEM  (FSCS) 


(FAS,  FSCS) 


In  late  CY  1996,  the  Army  identified  a  need  to  field  a  cavalry  vehicle  that 
provides  the  soldier  with  advanced  capability  to  look  forward  for  scouting.  As  a 
result,  the  US  Army  partnered  with  the  United  Kingdom  (UK)  to  develop  the 
Future  Scout  Cavalry  System  (FSCS).  The  intent  was  to  have  both  parties 
benefit  from  advanced  design  work  while  supplying  specific  subsystems. 
Although  the  Army  Transformation  effort  has  now  identified  the  Future  Combat 
System  as  the  platform  to  meet  the  requirements  and  FCS  has  replaced  the 
FSCS  concept,  the  initial  integration  of  SBA  into  FSCS  is  worth  reviewing  as  it 
exemplifies  positive  methods  in  which  to  employ  models  and  simulations. 

The  FSCS  is  operated  by  a  three-man  crew,  has  Micro  Unmanned  Aerial 
Vehicles  (MAVs),  unmanned  ground  operated  vehicles  noted  as  robots.  The 
United  Kingdom  is  responsible  for  the  Tactical  Reconnaissance  Armoured 
Combat  Equipment  Requirement  (TRACER).  Both  the  robots  as  well  as  the 
MAVs  are  designed  to  be  operated  as  forward  looking  devices  to  seek  out  enemy 
positions.  The  FSCS  also  contains  a  mast  that  can  rise  to  a  height  of  5  meters 
(FAS  -  FSCS,  2002). 

Advancement  of  the  FSCS  design  was  facilitated  by  a  simulation  called 

Close  Combat  Tactical  Trainer  (CCTT).  The  use  of  this  simulation  was  crucial  in 

providing  the  user  with  a  working  model  from  which  to  make  further  decisions 

regarding  design  or  doctrine.  CCTT  provides  personnel  with  different  interests 
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and  perspectives  -  the  soldier,  the  commander,  and  the  tester,  for  example  - 

with  the  capability  to  derive  diverse,  but  related,  information  from  the  item  being 

simulated.  For  example,  in  terms  of  a  FSCS,  the  soldier  would  have  a  system 

that  replicates  the  fidelity  of  the  combat  vehicle’s  operation  in  order  to  understand 

better  how  it  will  respond  to  different  scenarios.  The  commander  can  observe 

the  implementation  and  effectiveness  of  different  battle  tactics.  The  tester  can 

gather  data  related  to  the  strengths  and  weaknesses  of  the  overall  systems. 

The  FSCS  was  identified  as  one  of  the  SBA  Flagship  programs  at  the  right 

point  in  its  life  cycle  to  use  CCTT.  Since  the  FSCS  program  was  new,  the  use  of 

CCTT  seemed  ideal.  According  to  Truelove,  by  building  a  design  and  description 

that  are  digitized,  the  user  can  identify  flaws  as  the  development  progresses. 

Additionally,  the  program  will  use  the  SBA  concepts  repetitively  and  end  up  with 

embedded  training.  Finally,  the  program  can  use  virtual  prototypes  that  will 

identify  total  ownership  issues  (Truelove,  1999). 

CCTT  is  a  Distributed  Interactive  Simulation  (DIS)  meaning  that  users  can 

interact  with  the  simulation  from  different  workstations  (even  geographically 

separated)  at  the  same  time.  Dr.  Stuart  W.  Olson  published  a  paper  describing  a 

hypothetical  application  of  SBA  to  the  FSCS  program.  In  this  paper,  he 

addresses  how  SBA  can  be  implemented  across  program  phases  using  the 

simulation  tool  of  CCTT.  Personnel  are  trained  in  the  CCTT  using  both  a  full 

crew  simulator  as  well  as  a  command  post.  The  simulators  offer  such  reality  that 

trainees  can  fully  perform  their  normal  tasks  to  achieve  mission  success.  Also 

inserted  into  the  simulation  are  the  Semi  Automated  Force(s)  (SAF)  that  operate 

within  the  scenario  without  significant  human  intervention.  (Olson,  1998) 

Olson  goes  on  to  define  how  in  each  stage  the  information  derived  from 

the  simulation  can  be  utilized.  By  way  of  example,  Olson  suggests  that  in  the 

concept  phase,  CCTT  can  be  used  to  observe 

...combat  effectiveness,  information  flow,  tactics,  techniques,  and 
procedures,  command  and  control,  and  soldier  constraints  of  the 
concept  (Olson,  1998). 
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In  the  engineering  phase,  the  design  models  can  be  tested  with  CCTT  and 
improvements  made  accordingly.  Olson  goes  on  to  state  that  in  the  test  and 
evaluation  phase  that  the  amount  of  live  testing  can  be  reduced  through  the 
simulations.  Finally,  in  the  deployment  phase  of  this  concept,  the  advantages 
realized  in  the  early  phases  are  immediately  seen  in  this  stage  of  the  program. 
(Olson,  1998) 

B.  CRUSADER 


Forward  Vehicle:  Self-  Propelled  Howitzer  ( SPH 


(Army  Technology,  13  Nov  02) 
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Rear  Vehicle:  Resupply  Vehicle  (RSV) 


The  Crusader  program  was  cancelled  (in  May  2002)  because  the  concept 
did  not  fit  the  Army  Transformation  requirement.  The  concept  of  the  two  vehicles 
for  the  Crusader  was  noble,  but  the  practicality  of  the  overall  system  did  not  meet 
the  requirement  for  strategic  mobility.  Although  major  revisions  were  initiated, 
the  total  system  was  simply  too  large  to  be  transported  anywhere  in  the  world  in 
support  of  our  rapid  deployment  forces. 

The  Crusader  program  entered  Program  Definition  and  Risk  Reduction 
mid-year  1997.  This  system  was  comprised  of  two  vehicles  that  interlocked  on 
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the  field.  The  forward  vehicle  was  the  actual  platform  consisting  of  a  self- 
propelled  155mm  howitzer,  Modular  Ammunition  Charge  System  (MACS) 
propellant,  the  system  for  automatic  gun  positioning,  and  an  automated 
ammunition  handling  system.  The  forward  vehicle  was  designed  to  have  fire 
support  digital  communications  as  well  as  navigational  aids  to  support  the  crew  in 
field  mobility.  The  rear  vehicle  was  capable  of  maneuvering  to  the  forward 
vehicle  on  the  field  and  interlocking  with  it.  Once  coupling  had  been  achieved, 
automated  transferring  of  projectiles  and  propellants  would  take  place  (Cartwright 
and  Wallestad,  1998). 

The  US  Army  mandated  use  of  SBA  in  the  Crusader  program  as  one  of 
the  Flagship  programs.  The  rationale  for  this  decision  was  that  the  program 
could  identify  and  demonstrate  verification,  validation,  and  accreditation  (VV&A) 
of  modeling  and  simulation.  Additionally,  the  implementation  of  collaborative 
efforts  was  exploited  in  the  program.  For  example,  there  was  collaborative 
prototyping,  a  collaborative  environment  (also  known  as  Integrated  Digital 
Environment),  and  training  on  virtual  prototypes  (Truelove,  1999). 

The  application  of  SBA  in  the  Crusader  through  modeling  and  simulation, 
was  applied  throughout  its  development.  Use  of  modeling  and  simulation  proved 
to  be  effective  in  each  application  as  shown  in  the  selected  examples  that  follow. 

1.  Engineering  Design 

The  use  of  M&S  in  the  Crusader  engineering  design  was  a  consistent 
element  throughout  this  process.  According  to  Cartwright  and  Wallestad,  M&S 
applied  in  the  early  stages  of  design  identifies  errors  in  logic  as  well  as  gray 
areas  in  the  requirements.  Identifying  these  issues  prior  to  solidifying  the  design 
lessens  design  costs  as  well  as  reduces  time  through  the  program  development. 
The  authors  further  state  that  using  modeling  to  mirror  design  steps  offers 
efficiencies  not  otherwise  available.  Eventually,  the  design  becomes  a  “life-cycle 
model”  that  evolves  as  the  program  advances  (Cartwright  and  Wallestad,  1998). 
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2.  Modeling  and  Simulation  for  Test  and  Evaluation  (T&E) 

The  use  of  modeling  and  simulation  in  Test  and  Evaluation  is  crucial  to 
effective  management  of  a  program.  The  application  in  the  Crusader  program 
proved  to  be  no  different.  Although  use  or  planned  use  was  to  occur  in  all 
phases,  the  use  of  M&S  in  PDRR  was  to  be  more  extensive  than  in  either 
Engineering  and  Manufacturing  Development  (EMD)  or  Low  Rate  Initial 
Production  (LRIP).  The  authors  identified  four  objectives  for  use  of  Modeling  and 
Simulation  support.  These  are: 

•  To  facilitate  the  integration  and  test  of  Crusader  hardware  and 
software  products  at  the  subsystem,  element,  and  segment, 
levels; 

•  To  enable  item  checkout  and  debug  activities  at  component  and 
lower  levels; 

•  To  support  item  acceptance,  software  test,  and  hardware  test 
effort; 

•  To  provide  an  infrastructure  for  integration  and  interoperability  of 
external  models  and  simulations  used  in  system  level  testing 
and  analysis  (Cartwright  and  Wallestad,  1998). 

Through  the  use  of  M&S,  it  was  hoped  the  program  would  achieve  greater 
efficiency  in  its  testing  and  evaluation  elements.  Components,  subsystems,  and 
systems  would  be  verified,  while  still  at  a  point  where  design  changes  could 
easily  be  made. 


3.  M&S  for  Milestone  II 

The  use  of  modeling  and  simulation  in  this  program  to  support  Milestone  II 

was  significant.  According  to  Cartwright  and  Wallestad,  M&S  had  been  used  as 

...an  aid  in  test  design,  and  in  many  cases  as  a  modeled  or 
simulated  test  environment.  In  most  cases... real-time  simulations 
will  be  run  in  conjunction  with  the  ICE  (Integrated  Crusader 
Emulator)  at  UDLP  (United  Defense,  the  contractor)  to  dry  run  the 
test  in  order  to  verify  that  the  system,  as  configured  for  the  test,  is 
capable  of  attaining  a  firing  state  according  to  the  test  procedures 
(parentheses  added).  (Cartwright  and  Wallestad,  1998) 
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Modeling  and  simulation,  in  this  case,  provided  the  means  in  which  to  test 
the  simulation.  The  authors  went  on  to  state  that  actual  test  data  would  be 
measured  against  the  model  performance,  thereby  establishing  the  need,  if  any, 
for  improvements  or  isolating  unusual  characteristics  of  the  system.  Additionally, 
modeling  and  simulation  supported  wide-ranging  system  parameters  including 
range  (gunfire  dynamics),  accuracy  (Monte  Carlo  runs),  mobility  (vehicle 
dynamics),  and  crew  size  (Crew  Module  Demonstrator). 


C.  AH  -  64  APACHE  LONGBOW 


AH  -  64  Apache  Longbow 


(Army  Technology,  14  Nov  02) 
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The  Apache  helicopter  is  the  Army’s  attack  platform  to  be  used  in  close 
combat  and  to  penetrate  behind  lines  to  destroy  and  disrupt  enemy  forces.  The 
system  is  equipped  with  Hellfire  missiles  for  use  against  armored  enemy  forces, 
Hydra  70  (2.75”)  folding  fin  rockets  for  soft  targets,  and  30mm  cannon  rounds. 
Additionally,  the  system  has  an  optical  array  consisting  of  Day  TV,  Forward 
Looking  Infrared  (FUR)  detection,  and  a  color  magnified  imaging  system  (FAS  - 
Apache,  2002).  The  Longbow  variant  (UH-64D)  of  the  Apache  utilizes  a  mast- 
mounted  fire  control  radar  for  the  missile  (Hellfire)  system.  This  system  is 
deployable  for  use  in  all  conditions  regardless  of  time  of  day  or  visibility.  The 
Longbow  system  utilizes  digitization  to  acquire  up  to  128  targets  and  then 
prioritize  16  items  as  well  as  hand  off  targeting  data  to  other  attack  platforms  (US 
Army  TACOM,  2002). 

The  program  used  modeling  and  simulation  in  terms  of  providing 
simulators  to  compare  predicted  results  with  live  fire  testing.  In  this  case,  the 
program  used  simulations  of ... 

...target  acquisition  and  fly-out,  Hardware-in-the-Loop  (HWIL) 
testing  of  the  guidance  section,  low-speed  captive  flight  tests 
(LSCFT)  of  the  missile  seeker,  and  live  firings  at  moving  armored 
vehicles  (DOT&E,  1998). 

Therefore,  in  this  utilization  of  M&S,  the  Apache  Longbow  program  took 

action  to  measure  actual  results,  compared  to  those  derived  in  a  simulation. 

Additionally,  the  simulation  itself  was  tested  to  determine  its  reliability.  The 

application  of  simulations  attempted  to  forecast  what  to  examine  in  testing, 

should  results  vary.  As  stated  in  the  citation, 

...M&S  was  used  to  characterize  the  missile’s  performance  in  the 
LOBL-I  (Lock-On  Before  Launch  Inhibit))  mode  in  a  far  wider  range 
of  conditions  than  could  be  examined  just  using  field  test... The 
results  from  the  LSCFT  and  the  missile  firings  were  then  compared 
to  the  M&S  predictions  to  help  further  validate  the  simulation 
models  (DOT&E,  1998)  (parentheses  added). 

M&S,  in  these  Apache  cases  above,  was  used  to  both  forward  the 

program  and  reduce  costs  through  simulated  testing.  Additionally,  the 
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application  of  data  improved  system  performance.  Further  evidence  of  design 
considerations  is  presented  in  the  next  section  on  the  UH  -  60  Blackhawk 
Helicopter. 


D.  UH-  60  BLACKHAWK  HELICOPTER 


UH  -  60  Blackhawk 


Photo  taken  by:  USAF 


(Combat  Aircraft,  15  Nov  02) 


Another  Helicopter  platform  widely  used  in  the  US  military  is  the  UH-60 
Blackhawk  helicopter.  It  is  a  utility  aircraft  designed  to  be  operated  by  three  crew 
members  and  carry  a  fully  equipped  combat  squad  of  eleven.  The  platform  is 
qualified  to  carry  (sling  load)  a  105mm  howitzer  and  30  howitzer  rounds.  The 
helicopter  can  also  serve  in  roles  of  medical  (UH-60  MEDEVAC),  firefighting 
(UH-60  Firehawk),  electronics  counter  defense  (EH-60A),  and  target  acquisition 
(EH-60B)  (FAS  Blackhawk,  2002).  Currently,  the  Blackhawk  is  in  a  program 
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upgrade  (UH-60M)  status.  The  helicopter  is  undergoing  redesign  to  achieve 

increased  capabilities,  including  the  amount  of  weight  it  may  lift,  operating  range, 

and  survivability  (Edwards  and  Nikonchuk,  2001). 

Modeling  and  simulation  is  being  incorporated  into  this  program  as 

required  by  the  US  Army  directive,  Simulation  and  Acquisition  for  Acquisition, 

Requirements,  and  Training  (SMART).  According  to  the  authors,  the  intent  of 

SMART  in  the  program  is  to  prove  out  design  effectiveness  and  reduce  the  cost 

of  testing.  In  order  to  achieve  the  flexibility  in  the  hardware  design,  a 

reconfigurable  cockpit  was  developed  to  meet  the  design  specifications.  The 

program  intended  to  upgrade  the  flight  controls  through  a  new  design.  In  order  to 

test  the  design,  a  virtual  prototype  was  developed  and  tested  in  the 

reconfigurable  cockpit.  The  cockpit  design  was  handled  through  a  Computer 

Aided  Design  (CAD)  program,  which  allowed  the  personnel  to  view  different 

designs  in  a  direct  comparison  on  screen.  The  CAD  program  permitted  the  form, 

fit,  and  function  criteria  to  be  considered  and  incorporated.  Early  User 

Demonstration  (EUD)  will  comprise  three  events.  These  are: 

EUD1  will  use  CAD  and  computer-generated  imagery  to  facilitate 
user  and  designer  communication  and  analysis  during  risk 
reduction  and  prior  to  PDR.  EUD1  allows  pilots  (users),  designers, 
and  PMO  representatives  to  identify  potential  user  issues  and 
design  solutions  based  on  current  configurations.  EUD1  will  also 
provide  an  opportunity  to  define  the  metrics  necessary  to  measure 
situational  awareness  (SA)  resulting  from  information  presented  to 
the  pilot  (Event  1)\  establish  measure  of  effectiveness/performance 
for  future  SA  design  and  analysis  activities  (Event  2)\  and  facilitate 
initial  human  factors  engineering  of  candidate  instrument  panel 
configurations  (Event  3)  (Edwards  and  Nikonchuck,  2001) 
(parenthesis  added) 


Again,  SBA  through  modeling  and  simulation,  is  being  utilized  to 
effectively  improve  system  performance.  The  ability  to  immediately  reconfigure  a 
helicopter  cockpit  based  upon  design  comparison  is  a  powerful  tool  for  the 
program  manager. 
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E.  ARMY  USE  OF  SURVEY  DATA 

The  preceding  four  cases  define  the  use  of  Simulation  Based  Acquisition 
methods  through  concrete  examples.  In  all  of  the  cases,  the  program  managers, 
and  the  acquisition  chain,  recognized  the  potential  for  positive  program  impacts 
by  employing  these  methods.  Examples  of  Army  use,  that  mirror  survey  data 
from  Navy  acquisitions,  include  simulation  of  hardware  environments,  prototype 
development,  use  of  CAD/CAM,  software  verification,  and  platform  qualification. 
It  should  be  noted  that  the  programs  exemplified  above  reflect  four  very  high 
profile  programs  and  therefore  the  SBA  methods  employed  are  numerous. 
Accordingly,  it  is  logical  that  the  Army  cases  are  significant  in  their  replication  of 
Navy  uses. 

Throughout  this  chapter,  the  paper  has  identified  different  ways  in  which 
program  managers  can  utilize  modeling  and  simulation  to  achieve  program 
success,  build  prototypes,  effectively  conduct  test,  and  validate  simulations. 
Each  of  the  programs  benefited  from  these  applications  of  modeling  or 
simulations  and  the  system  designs  became  dynamic,  oftentimes  at  a  point  in  the 
program  where  such  measures  served  the  greatest  good. 
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VI.  CONCLUSIONS  AND 
RECOMMENDATIONS 


The  intent  of  this  thesis  paper  was  to  answer  the  Primary  Thesis  Question, 
“Are  there  specific  times  within  the  life  cycle  of  a  major  program  when  it  is  in  the 
best  economic  interest  of  the  Program  Manager  to  use  the  methods  of  Simulation 
Based  Acquisition?” 

In  order  to  answer  this  question,  the  study  reviewed  guidance  offered  at 
the  Department  and  Service  levels  within  the  Department  of  Defense.  Case 
studies  were  presented  that  illustrated  Simulation  Based  Acquisition  (SBA)  used 
in  current  or  previous  programs.  Additionally,  a  survey  was  sent  to  Navy 
program  managers  for  input  on  their  SBA  use  and  successes. 

This  concluding  chapter  will  review  the  significant  observations  from 
earlier  chapters,  provide  conclusions,  and  make  recommendations  for  future 
research. 

A.  SIGNIFICANT  OBSERVATIONS 

1.  Introduction 

In  the  thesis  introductory  chapter,  the  paper  presented  a  definition  of 
Simulation  Based  Acquisition  that  identified  the  use  by  Government  and  industry, 
in  a  collaborative  environment,  and  the  simulation  technologies  currently 
available.  The  use  should  be  across  phases  and  programs.  The  intent  is  to 
reduce  the  consumption  of  resources,  including  time,  and  risk  while  increasing 
the  quality  of  the  program.  The  suggested  manner  in  which  to  achieve  this  is 
through  Integrated  Product  and  Process  Development  (IPPD)  in  programs.  SBA 
should  not  be  considered  to  replace  common  sense  and  required  activities. 
Essentially,  the  prudent  program  manager  should  incorporate  SBA  where  it 
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serves  the  interests  of  the  Service  and  the  taxpayer,  through  the  available 
technologies,  as  a  part  of  the  program  acquisition  cycle. 

Another  point  identified  in  the  introductory  chapter  is  the  new  paradigm  of 
testing.  The  standard  testing  progression  had  been  test-fix-test.  Using  SBA,  the 
methodology  shifts  to  model-simulate-test  with  the  test  results  driven  back  into 
the  model.  This  strategy  should  be  explicit  in  every  program’s  TEMP  and 
followed  zealously. 

Additionally,  the  chapter  presented  information  from  the  old  DoDD  5000 
series  instructing  program  managers  to  utilize  Modeling  and  Simulation  within 
their  programs.  The  purpose  of  this  was  to  reduce  time,  resources,  and  risk. 
These  are  the  elements  that  were  present  in  the  earlier  definition  of  SBA. 
Clearly,  the  intent  of  the  guidance  and  initial  SBA  documentation  addressed  the 
need  to  reduce  program  cost  elements  while  increasing  the  quality  of  the 
systems.  The  interim  guidance  promulgated  in  November  2002  by  the 
DEPSECDEF  does  not  change  the  intent. 

There  are  numerous  methods  of  modeling  and  simulation  that  program 
managers  can  employ  in  their  programs.  Examples  include  three-dimensional 
design  capabilities  with  input  from  all  stakeholders  in  real  time;  distributed 
interactive  simulations  in  which  virtual  forces  are  set  in  motion  from  wide  (real 
life)  geographical  locations;  virtual  training  for  the  soldier;  and,  using  simulators 
to  replicate  live  fire  tests.  Before  simulations  are  accepted,  actual  testing  or 
other  verification  means  must  be  used  to  validate  the  simulation. 

2.  Background 

SBA  was  defined  to  include  three  goals  consisting  of  the  following:  to 
reduce  time,  risk,  and  resources;  to  increase  the  quality  of  programs;  and,  to 
enable  the  use  of  IPPDs  in  the  life  cycle.  However,  program  managers  are 
encouraged  not  to  rely  upon  SBA  as  a  replacement  for  common  sense  or  quality 
systems  engineering.  The  Department  of  Defense  established  in  1995  that 
programs  need  to  use  modeling  and  simulation  more  extensively.  This  would 
decrease  the  duration  of  testing  and  change  the  testing  process  into  modeling, 
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simulation,  testing,  and  then  incorporating  the  results  into  the  model.  In  1998, 
DoD  stated  that  the  military  needed  to  develop  new  major  systems,  support 
them,  and  do  it  at  a  lower  cost.  Additional  guidance  came  out  of  the  Department 
of  Defense  in  the  form  of  DoDD  5000.1  and  the  USD  memorandum.  Both  of 
these  documents  discuss  the  control  of  costs  and  resources  to  manage 
programs  effectively.  The  Interim  guidance  that  has  replaced  the  5000  series 
details  the  implementation  of  cost  realism,  sharing,  and  affordability  in  programs. 
Private  industry  encourages  the  use  of  SBA  and  suggests  that  systems 
engineering  and  better  design  data  are  the  key  elements  in  the  SBA  process. 

3.  Guidance 

The  Department  of  Defense  and  each  of  the  Services  have  all  put  out 
guidance  on  modeling  and  simulation.  Each  Service  has  established  a  Modeling 
and  Simulation  plan  as  well  as  policy  for  M&S.  Additionally,  the  Services  have 
promulgated  their  information  on  the  Internet  in  a  variety  of  styles.  However,  the 
question  is  raised  of  how  often  the  guidance  and  plans  are  updated.  Additionally, 
the  research  proved  that  there  is  not  a  single  specific  Internet  site  for  SBA 
throughout  the  Services. 

4.  Survey  Data  and  Results 

The  thesis  survey  provided  results  that  were  quite  divergent  in  nature.  A 
significant  feature  of  the  survey  was  that  programs  across  all  ACAT  levels  from 
the  Navy  were  requested  to  complete  the  questionnaire.  A  reasonable  response 
was  received  and  at  least  four  significant  uses  of  SBA  were  found  in  the  DON 
programs.  These  were: 

Computer  Modeling  -  The  respondents  indicated  that  computer 
modeling  was  effective  in  reducing  testing  as  well  as  making  design  changes  in 
structural  or  form,  fit,  and  function  issues. 

CAD/CAM  -  CAD  and  CAM  were  used  as  an  aspect  of  SBA.  Additionally, 
the  engineers  looked  for  existing  M&S  that  could  be  applied  to  their  program. 
One  program  established  one  manager  for  M&S  PDRR  (now  Concept  and 

Technology  Development  and  System  Development  and  Demonstration). 
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Virtual  Prototyping  -  In  this  area,  prototyping  was  used  to  provide 
database  access  to  both  Government  and  industry  personnel.  Prototyping  was 
used  to  reduce  the  amount  of  rework  in  a  system  thereby  saving  man-hours. 
Although  not  discussed  by  survey  respondents,  prototyping  can  also  be  used  for 
risk  management.  The  USMC  Advanced  Amphibious  Assault  Vehicle  (AAAV) 
uses  database  management  for  risk  reduction. 

Production  Data  -  This  data  was  used  in  the  EMD  phase  (now  primarily 
System  Demonstration  and  Production  Readiness  and  LRIP)  to  update  the 
design  through  high  fidelity  representations. 

4.  Case  Studies 

Four  programs  were  presented  to  exemplify  how  SBA  can  be  applied  to  a 
program  or  system.  The  Future  Scout  Cavalry  System  (FSCS),  although 
canceled,  demonstrated  the  power  of  the  Close  Combat  Tactical  Trainer.  This  is 
a  trainer  that  puts  the  user  into  a  virtual  environment.  From  such  training,  the 
soldier  can  get  close  to  real  life  engagement  of  the  system.  The  tester  can 
review  how  the  system  behaves  and  reacts.  The  commander  can  observe  the 
strengths  and  weaknesses  of  the  system  in  battle  and  determine  tactical 
maneuvers  for  use  by  forces.  By  inserting  SBA  into  the  Crusader  early  in  its 
acquisition  cycle,  the  program  manager  was  able  to  change  the  design  easily. 
Additionally,  70%  of  the  downstream  logistics  cost  is  affected  by  design  factors 
normally  solidified  early  in  the  life  cycle.  By  changing  design  for  future  logistics, 
the  program  manager  can  realize  a  significant  savings  for  the  fielded  systems 
when  operating  and  support  budgets  are  typically  limited. 

The  Crusader  program,  also  cancelled,  implemented  SBA  in  its 
developmental  efforts.  Although  modeling  and  simulation  had  been  used  or 
planned  for  use  in  many  areas,  there  were  three  specific  areas  for  the  Crusader 
discussed  in  Chapter  4.  These  were  Engineering  Design,  Test  and  Evaluation 
Support,  and  Milestone  II  support.  Several  examples  of  how  M&S  was  expected 
to  benefit  Crusader  include  verification  of  the  models  (as  compared  to  live  fire 
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testing);  testing  of  the  system  and  subsystems;  and,  providing  a  structure  for 
demonstrating  interoperability. 

The  Apache  Longbow  used  M&S  for  test  system  performance  in 
parameters  that  were  not  easily  replicated  in  live  fire  testing,  while  the  Blackhawk 
program  used  M&S  in  conjunction  with  a  reconfigurable  cockpit  to  expedite 
design  changes. 

The  four  cases  provided  in  this  study  exemplify  the  wide  spectrum  of  uses 
of  SBA  and,  in  some  cases,  modeling  and  simulation.  Although  these  programs 
are  highly  visible  (or  were  so  in  the  cases  of  FSCS  and  Crusader),  the  use  of 
SBA  in  lower  level  ACAT  programs  should  not  be  overlooked. 

B.  CONCLUSIONS 

Through  a  review  of  current  guidance,  an  analysis  of  data  derived  from  a 
survey,  and  an  examination  of  case  studies,  this  thesis  has  identified  different 
uses  of  Simulation  Based  Acquisition.  Significant  conclusions  may  be  defined 
as  follows: 

1.  Guidance  and  Accessibility 

The  guidance  has  provided  a  roadmap,  albeit  patchwork  in  nature,  of  the 
use  of  SBA  in  programs.  It  should  also  be  noted  that  guidance  is  dynamic  and 
changes  over  time.  Accordingly,  it  makes  sense  to  have  a  central  Department  of 
Defense  Internet  site  for  SBA,  its  uses,  Service  guidance  and  policy,  and  lessons 
learned.  As  guidance  is  revised  and  issued,  the  Internet  site  would  be  updated 
to  stay  current.  The  hyperlinks  on  the  central  site  would  drive  down  to  Service 
sites,  which  need  to  have  links  to  the  DoD  site. 

2.  Funding  and  Return  on  Investment 

One  significant  finding  through  the  survey  was  that  funding  typically 
occurred  during  the  PDRR  or  EMD  phases  in  the  acquisition  life  cycle.  The  uses 
include  design  efforts  to  reduce  risk,  cost,  requirements  development,  platform 
qualification,  and  product  testing.  However,  the  program  manager  should  plan 


59 


for  the  cost  savings  that  SBA  will  return  throughout  the  entire  program.  Of  all  the 
survey  respondents,  when  posed  the  question  (of  planned  cost  savings),  only 
one  answered  in  the  affirmative  that  costs  savings  were  planned  through  SBA 
use.  This  is  significant  in  that  slightly  more  than  half  of  the  respondents  (who 
also  stated  that  the  program  had  planned  investment  in  SBA)  replied  that  they 
were  able  to  identify  cost  avoidance  though  the  use  of  SBA.  Therefore,  program 
managers  should  plan  on  using  SBA  methods  at  all  points  in  the  program  to 
achieve  some  measure  of  return  on  investment. 

3.  Missing  Uses 

The  survey  provided  information  on  how  SBA  was  being  used,  but  the 
responses  also  demonstrated  issues  that  were  omitted.  For  example,  there  were 
only  two  respondents  that  discussed  the  use  of  SBA  in  the  first  stage  of  the 
acquisition  life  cycle  (Concept  Exploration).  Furthermore,  there  was  no  mention 
by  respondents  of  design  interfaces  between  developers,  contractors,  and  users. 
Concept  Exploration  is  the  most  significant  area  in  which  to  employ  SBA  to 
achieve  program  savings  downstream.  Bringing  the  users,  developers,  and 
contractors  together  during  CE  is  crucial  for  resolution  of  design  issues.  It  was 
mentioned  in  Chapter  I  that  by  the  time  that  the  program  hit  Milestone  I  (under 
the  traditionally  acquisition  life  cycle  timeline)  that  70  percent  of  the  eventual 
O&S  costs  had  been  locked  in.  By  early  involvement  of  these  parties,  the 
program  can  achieve  significant  savings  and  benefits  in  the  future.  Additionally, 
there  was  little  discussion  of  use  of  SBA  in  place  of  live  fire  testing.  The  ability 
for  SBA  to  replicate  live  fire  testing  while  lowering  costs  is  significant  to  the 
program  and  should  be  used  as  frequently  as  possible.  Finally,  narrative 
responses  to  the  survey  did  not  indicate  to  what  degree  future  SBA  efforts  would 
be  made.  SBA  is  an  initiative  that  is  applicable  to  all  phases  of  a  program  and 
should  be  used  accordingly.  It  is  most  effective  when  used  in  the  early  stages, 
but  SBA  may  still  be  utilized  throughout  Production  as  well. 
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4.  Simulations 

The  Future  Scout  Cavalry  System  (FSCS)  case  study  identified  the  use  of 
simulations  in  the  program  by  employing  the  Close  Combat  Tactical  Trainer 
(CCTT).  The  program  used  the  simulation  to  identify  design  flaws  and  to  provide 
embedded  training.  Additionally,  the  CCTT  virtual  prototypes  were  described  as 
being  a  tool  that  the  program  manager  could  use  to  recognize  ownership  issues 
for  the  future. 

Program  managers  should  employ  the  use  of  simulations  within  the 
bounds  of  technical  feasibility,  required  time  lines,  and  good  economic  judgment 
to  achieve  design  fidelity,  risk  management,  ownership  costs,  training 
opportunities,  live  fire  replications,  and  prototyping.  The  implementation  of  this 
tool  provides  opportunities  to  the  program  manager  for  thinking  laterally  instead 
of  resorting  to  rigid  product  development. 

5.  Prototyping 

Although  prototyping  can  easily  occur  digitally  with  reviews  by  users  and 
engineers,  there  are  higher  levels  of  prototyping  that  may  take  place.  In  the 
Blackhawk  program,  design  configuration  is  done  normally  through  virtual 
prototyping.  However,  the  program  has  also  developed  a  physical  cockpit  that 
can  be  reconfigured  as  the  virtual  design  is  manipulated.  The  immediate  benefit 
that  can  be  realized  is  that  the  program  can  discern  the  impact  of  design 
changes  upon  the  cockpit.  But  long-term  benefits  may  be  derived  also.  Direct 
costs  are  reduced  through  less  labor  and  material  used.  Users  can  build  in 
design  features  for  future  hardware,  in  process,  but  not  yet  available  (next 
generation  of  communication  equipment  for  example).  Additionally,  logisticians 
can  use  the  current  configurations  to  begin  efforts  for  future  considerations  and 
adjust  as  the  design  changes.  Ultimately,  it  makes  sense  for  any  program 
manager  to  use  reconfigurable  prototypes  wherever  possible  to  advance  the 
program  and  achieve  design  and  cost  benefits. 
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6.  Economic  Breakpoints 

The  earlier  discussion  in  points  2  through  5  above  address  the  use  of  SBA 
in  programs.  The  thesis,  through  use  of  the  survey  and  research,  was  unable  to 
determine  a  level  of  funding  at  which  one  could  decide  to  employ  SBA 
techniques.  However,  the  guidance,  data,  and  case  studies  were  able  to 
establish  that  SBA  in  multiple  forms  can  be  applied  across  program  phases  to 
benefit  the  program.  The  program  manager  should  use  SBA  as  early  as 
possible,  but  that  does  not  preclude  use  in  later  stages  especially  after  the 
design  is  well  fixed. 

C.  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

The  intent  of  this  thesis  was  to  define  specific  breakpoints  in  military 
programs  in  which  to  employ  SBA  methods.  Although  breakpoints  could  not  be 
identified,  there  were  issues  that  could  be  pursued  in  future  research.  These 
include: 

•  A  survey  of  ACAT  level  I  programs  across  all  Services  that  examine 
more  thoroughly  the  funding  aspects  of  SBA. 

•  A  comparative  examination  of  two  specific  programs  in  which  one 
used  SBA  successfully  in  all  phases  and  methods  and  the  other 
underutilized  SBA. 

•  A  study  of  future  SBA  direction  as  set  by  DoD  and  the  Services 
while  working  closely  with  the  lead  individuals  in  each  area. 
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Figure  1.  A  Comparison  Between  the  5000  Series  Timeline  and 

the  Revised  Model 
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Figure  2.  Hierarchy  of  Modeling  and  Simulation 
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Figure  3.  M&S  Building  Blocks  Supporting  the  Areas  of 
Assessment ,  Training ,  and  Acquisition 
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(HQ  USAF,  1995) 

Figure  4.  Modeling  and  Simulation  Hierarchy 
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Model  Name 

Type 

ASOAR 

Achieving  a  System  Operational 

Availability  Requirement 

Readiness 

SESAME 

Selected  Essential-Item  Stock  for 
Availability  Method 

Readiness 

COMPASS 

Computerized  Optimization  Model  for 
Predicting  and  Analyzing  Support 
Structures 

Readiness  and 
Cost 

ACEIT 

Automated  Cost  Estimating  Integrated 
Tools 

Cost 

LCET 

Logistics  Cost  Estimating  Tool 

Cost 

(From  Price,  2002) 


Table  1.  Linked  /  Integrated  Army  Models 
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ACAT  Level 

SBA  as  %  of 
Acquisition  Cost 

Cost 

Avoidance  as 
%of 

Acquisition 

Cost 

1 

0.70 

2.60 

2 

0.20 

0.00 

2 

2.60 

1.70 

3 

1.00 

1.50 

3 

1.10 

0.00 

3 

1.40 

0.00 

3 

2.00 

2.60 

3 

2.90 

0.00 

3 

2.90 

0.00 

3 

4.40 

0.00 

3 

8.10 

0.00 

3 

11.30 

0.00 

3 

11.30 

0.00 

3 

19.80 

0.00 

4 

0.10 

0.20 

4 

1.00 

1.90 

4 

1.10 

0.00 

4 

4.60 

0.90 

Table  2.  Comparison  of  Projected  SBA  Cost  to  Anticipated  Cost 

Avoidance 
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